Feed aggregator

Manifesto: Introducing the Chatbot API module for Drupal 8

Drupal Planet -

Conversational interfaces powered by artificial intelligences are rapidly growing in popularity for delivering more natural-feeling, convenient experiences for consumers. We’re very excited to announce the new Chatbot API module for Drupal 8, which helps surface content via bots and assistants without having to write reams of code for each AI you want to use. Voice. Continue reading...

InternetDevels: The productive August: Drupal news wrap-up for 08/2017

Drupal Planet -

Greetings to everyone! It looks like “8” is a lucky number and 8/2017 is a lucky month for drupalers. By taking a little extra energy from the sun (which is pretty environmentally friendly), the Drupal community has made so many awesome things! It feels like yesterday that we offered you the July 2017 Drupal news summary, and now we’re moving on to the wrap-up of the hot and productive August 2017.

Read more

CodeSOD: Gotta Get 'Em All

The Daily WTF -

LINQ brings functional programming and loads of syntactic sugar to .NET languages. It’s a nice feature, although as James points out, it helps if your fellow developers have even the slightest clue about what they’re doing.

// some validation checking var retrieveDocIdList = this.storedDocumentManager.GetAllForClientNotRetrieved(client.Id).Select(x => x.Id.ToString(CultureInfo.InvariantCulture)).ToList(); retrieveDocIdList.ForEach(id => { var storedDoc = this.storedDocumentManager.Get(int.Parse(id)) // do some other stuff with the doc });

James writes:

The code snippet is somewhat paraphrased because the actual code is poorly formatted and full of junk, but this is the main point.
It seems to be a requirement that previous developers leave weird code behind with no documentation or comments explaining what they were thinking at the time.

Well, first, “poorly formatted and full of junk” is our stock-in-trade, but we do appreciate the focus on the main WTF. Let’s see if we can piece together what the developers were thinking.

If you’ve read an article here before, your eyes almost certainly will catch the x.Id.ToString and the int.Parse(id) calls. Right off the bat, you know something’s fishy. But let’s walk through it.

this.storedDocumentManager.GetAllForClientNotRetrieved(client.Id) returns a list of all the documents that have not beeen loaded from the database. Now, by default, this is equivalent to a SELECT *, so instead of getting all that data, they pick off just the IDs as a string in the Select call.

Now, they have a list of IDs of documents that they don’t have loaded. So now, they can take each ID, and in a ForEach call… fetch the entire document from the database.

Well, that’s what it does, but what were they thinking? We may never know, but at a guess, someone knew that “Select star bad, select specific fields”, and then they just applied that knowledge without any further thought. The other possibility is that the team of developers wrote individual lines without talking to anyone else, and then just kinda mashed it together without understanding how it worked.

James replaced the entire thing:

foreach (var storedDoc in this.storedDocumentManager.GetAllForClientNotRetrieved(client.Id)) { //do some other stuff with the doc } hljs.initHighlightingOnLoad(); [Advertisement] Incrementally adopt DevOps best practices with BuildMaster, ProGet and Otter, creating a robust, secure, scalable, and reliable DevOps toolchain.

[$] CPU frequency governors and remote callbacks

LWN Headlines -

The kernel's CPU-frequency ("cpufreq") governors are charged with picking an operating frequency for each processor that minimizes power use while maintaining an adequate level of performance as determined by the current policy. These governors normally run locally, with each CPU handling its own frequency management. The 4.14 kernel release, though, will enable the CPU-frequency governors to control the frequency of any CPU in the system if the architecture permits, a change that should improve the performance of the system overall.

Classic WTF: #include "pascal.h"

The Daily WTF -

It's Labor Day in the US, where to honor workers, some people get a day off, but retail stores are open with loads of sales. We're reaching back to the old days of 2004 for this one. -- Remy

Ludwig Von Anon sent in some code from the UI component of a large, multi-platform system he has the pleasure of working on. At first glance, the code didn't seem all too bad ...

procedure SelectFontIntoDC(Integer a) begin
 declare fonthandle fh;
 if (gRedraw is not false) then begin
   fh = CreateFontIndirect(gDC);
   SelectObject(gDC, fh);
   DeleteObject(fh);
 end;
end;

Seems fairly normal, right? Certainly nothing that meets our ... standards. Of course, when you factor in the name of the codefile (which ends in ".c") and the header included throughout the entire project ("pascal.h"), I think it becomes pretty apparent that we're entering Whiskey Tango Foxtrot country:

#define procedure void
#define then
#define is
#define not !=
#define begin {
#define end }

Yeeeeeeeee Haw!  Sorry, just can't get enough of Mr. Burleson.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!

Error'd: Thresholds Were Made to be Broken

The Daily WTF -

Dima R. wrote, "Running out of space on this old XP machine. I know, I'll just uninstall TurboTax 2014!"

 

George wrote, "Encountered this while filling out my Australian tax return via the ATO's website, but it's more like like an ATUH-OH if you ask me!"

 

"Their motto should be 'We put your health first...and your security last!'" Vladimir B. writes.

 

"Well, since jeans cover the top of your lap, it's maybe, kind of a laptop," wrote Mike R.

 

"I guess on the bright side, at least PayPal knows how when and how much I paid to Undefined," wrote Travis M.

 

"I didn't realize that screaming in my resume was a requirement," Shyam writes.

 

The Sketch editor worked great! That is, until someone decided not to look at filenames," writes Jeff K.

 

[Advertisement] Otter enables DevOps best practices by providing a visual, dynamic, and intuitive UI that shows, at-a-glance, the configuration state of all your servers. Find out more and download today!

Attention to Detail

The Daily WTF -

The exotic and exciting life of the world-traveling contractor wasn’t exactly what Angie had been expecting. It mostly meant living in a dreary apartment on the outskirts of some city in a short drive from an industrial park where she’d go to try and keep 30-year old C code and their new ERP from fighting to the death. Six months later, she’d be off to the same apartment near the same industrial park in a different country.

When the crash came, it came hard. Hard enough that Angie ditched IT and got a temp job working in a customer service call-center for a greeting card company. She wasn’t exactly the best person on the phone, and nobody was giving her stellar marks for her cheerful demeanor during her quarterly review.

What her boss did notice though, is that when she did order entry, it was accurate. This was surprisingly a big deal, because the number of orders with typos coming from the other reps was remarkable. “We really appreciate your attention to detail,” he said. He offered to make the temp job permanent and start working on some career advancement within the company.

Angie took it. Over months, she spent less time on the phone, and more time putting her attention to detail to work: cleaning up order entry processes. Since Angie was a developer, she wrote some scripts to streamline the process and shared them with her team. Now, her boss was praising her attention to detail and her initiative.

Within a few months, the dev team offered to bring her aboard. The salary bump was nothing to sneeze at, and they didn’t care that she knew C and Java and Ruby, but not their language of choice- C#. So she moved departments, and started working for Liam.

Liam was the lead architect, and back in the early days, he was their only developer. Most of the software was home-grown extensions to their ERP, or their CRM. Since engraving printing plates was itself pretty complicated, he’d whipped up a program that could generate output to control the engraving system that made printing plates.

Like a lot of smaller software teams in large companies that don’t view software as a priority, the code quality was… special. For any given program, most of the code was in one gigantic do-everything class, or worse, just in the main method. Version control was naming files “Foo.cs.old” or “Foo.cs.dontuse”, and release management was hitting “build” and copying the output to a network share.

Liam, as the lead architect, didn’t want Angie wasting her time on the “big picture” stuff. “You’ve got such a great attention to detail,” he said. This meant she ended up being the SQL and regular expression expert who also tested the programs (often in production, because that was the only way to test). The result was far fewer bugs, fewer accidents from testing in production, and happier end users.

The work was messy, but it wasn’t hard, and the card company didn’t really expect a lot from their software team. Angie appreciated sleeping in the same bed every night, and actually having a social life.

Late on a Friday, the head of the company’s charity efforts burst into their cube-farm. The charity team had just run a major fund drive, and now needed to send out custom “Thank You” cards. There was only problem- the template they used (which drove Liam’s program to control the engraver) needed space for one additional line of text. “We need to get these running on the presses tomorrow so we can send them out next week!”

It was a four-alarm, hair-on-fire crisis, according to the charity chief. They were more than happy to provide dinner for the team who worked late, but it needed to be done. Since Angie was “detail oriented”, she drew the short straw, but she needed Liam’s help to get the changes made. “I don’t understand any of this code, and I can’t follow the logic.”

“Well,” Liam said, “I can’t say that I do, either.”

“But you wrote it!”

“Okay, yeah, let’s take a look.”

Over some surprisingly high-quality Thai takeout, Liam and Angie did their best to trace through the logic of the code, understanding how it consumed the template and converted it into something the engraver could understand. Because there was a lot of code-reuse by copy-and-paste, they identified three places that needed changes.

“Are you sure that’s it?” Angie asked.

“Yeah, absolutely.”

“Okay… but how do we test this?”

“The only way to test it is to send it to an engraver.”

“Okay, well… let’s go through it again and make sure it’s right,” Angie suggested.

It was rubbing up against 10PM, and Liam had enough of that. “Let’s just run it and get the plate engraved. I’ve done this sort of thing a bunch, it’ll be fine.”

It wasn’t. By the time anyone had noticed, however, the plate was already off to the presses. The resulting run cost the company $10,000 in materials, and delayed the sending of the “thank you” cards by three days, which the charity team warned could seriously hurt their charity efforts in the future. The big bosses stormed into the development team’s office, demanding: “Who’s responsible for this?”

The bus was coming, and Liam was ready to throw Angie right in front of it. “We assigned that project to Angie,” he said. She was escorted out of the office that day, and on her exit paperwork, the reason for termination was "insufficient attention to detail".

[Advertisement] BuildMaster integrates with an ever-growing list of tools to automate and facilitate everything from continuous integration to database change scripts to production deployments. Interested? Learn more about BuildMaster!

CodeSOD: Take a Byte of a Nibble

The Daily WTF -

Imagine, if you will, that you have 64-bits of data. From this 64-bits of data, you need to extract a nibble, which contains the value that you care about. Now, I’m sure you’re imagining an integer with some bitmasks to extract the data, which is a perfectly sane approach.

Tomasz inherited some code from his company’s German office. It took the approach of taking the 64-bits and storing the 64-bits in an eight element byte array. Then, it extracted the values from that array with code looking like this:

if ((app.xuc_Special_TRX[EMV_ADK_REFUND_BYTE] EMV_ADK_REFUND_NIBBLE) == EMV_ADK_REFUND_NO) { ... }

“What’s this?” Tomasz wondered. “This code couldn’t possibly compile… not unless the operator is hidden in the macro…”

#define EMV_ADK_EMV_ADK_MANUAL_REVERSAL_BYTE 0 ///< byte for configuration of manual reversal #define EMV_ADK_MANUAL_REVERSAL_NIBBLE >> 4 & 0x0F ///< nibble for configuration of manual reversal #define EMV_ADK_REFUND_BYTE 0 ///< byte for configuration of refund #define EMV_ADK_REFUND_NIBBLE & 0x0F ///< nibble for configuration of refund #define EMV_ADK_EMV_ADK_RESERVATION_BYTE 1 ///< byte for configuration of reservation #define EMV_ADK_RESERVATION_NIBBLE >> 4 & 0x0F ///< nibble for configuration of reservation #define EMV_ADK_TIP_BYTE 1 ///< byte for configuration of tip (gratuity) #define EMV_ADK_TIP_NIBBLE & 0x0F ///< nibble for configuration of tip (gratuity) #define EMV_ADK_REFERRAL_BYTE 2 ///< byte for configuration of referral @n not used for contactless #define EMV_ADK_REFERRAL_NIBBLE >> 4 & 0x0F ///< nibble for configuration of referral @n not used for contactless #define EMV_ADK_VOICEAUT_BYTE 2 ///< byte for configuration of voice authorization @n not used for contactless #define EMV_ADK_VOICEAUT_NIBBLE & 0x0F ///< nibble for configuration of voice authorization @n not used for contactless #define EMV_ADK_RFU_MODE_BYTE 3 ///< byte RFU #define EMV_ADK_RFU_MODE_NIBBLE >> 4 & 0x0F ///< nibble RFU #define EMV_ADK_FALLBACK_AFTER_CVM_BYTE 3 ///< byte for configuration of "fallback to magstripe after start of cardholder verification or early PIN entry allowed" @n not used for contactless #define EMV_ADK_FALLBACK_AFTER_CVM_NIBBLE & 0x0F ///< nibble for configuration of "fallback to magstripe after start of cardholder verification or early PIN entry allowed" @n not used for contactless #define EMV_ADK_IGNORE_CARD_ERROR_BYTE 4 ///< byte for configuration of "ignore card error after issuer authorization" #define EMV_ADK_IGNORE_CARD_ERROR_NIBBLE >> 4 & 0x0F ///< nibble for configuration of "ignore card error after issuer authorization"

The structure of the surrounding code makes it clear that there's no reason to store this data in an array- the offending developer chose an array over an integer. Further, we know that they know how to use bitmask operations and bitshift operations, so there's no reason to have written this code.

Since this is an array of bytes, each array index contains two values. Thus, EMV_ADK_REFUND_NIBBLE is but a bitshift away from EMV_ADK_MANUAL_RREVERSAL_NIBBLE. Just don’t try and understand what these macros do frrom where they’re used, and you’ll be fine. Oh, and also don’t accidentially mismatch them- if you use the wrong EMV_ADK_*_BYTE with the wrong EMV_ADK_*_NIBBLE you’ll get the wrong data back without any error. If you don’t do those two things, you’ll be fine.

hljs.initHighlightingOnLoad(); [Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!

Paying Taxes on Technical Debt

The Daily WTF -

In the U.S., individuals are expected to file federal and state tax returns once a year by April 15. The tax forms are quite complicated, and have all sorts of sub-forms and schedules to support and detail the numbers on the main form. The tax code of the U.S. is approximately 74,000 pages of special cases.

For many items, the same data needs to be entered on multiple forms, usually as the starting point for different calculations that depend upon the same information; these are duplicated again on both federal and state returns. It follows that tax preparation software needs to put the relevant numbers in all the places that they are needed.

Why? Because if it doesn't, the preparer needs to manually copy numbers to each of several places, leading to all sorts of omission/accidental editing issues. Just to make it needlessly complicated interesting, in many cases, the numbers need to be transformed via some formula before copying, and the formulas vary from form to form, and from state to state. Complicating this process is the fact that tax forms stating your earnings from employers and financial institutions can be re-issued with new values if tax laws are changed too close to the end of the year. This means that new numbers need to be entered, propagated, calculated and checked on your tax forms, usually after they were initially prepared.

If you're preparing your own tax returns, then they probably aren't all that complicated, and there isn't too much copy/pasting to be done. If you have multiple businesses in different states/countries, multiple properties, use multiple banks/brokerages, etc., then there are many, many forms and worksheets detailing the same information, leading to a whole lot of copy/pasting of data. When the complexity of what must be prepared gets beyond a certain point, you go to a professional preparer who will just enter your information once, and the very expensive professional tax software will put the numbers on all the relevant forms automatically. This way, it only takes a few minutes to enter the data, hit calc and a mountain of completed tax forms spews forth from the printer.

Julie is a highly experienced accountant, and was partnering with a large, national chain of tax preparers that used their own in-house tax preparation software. They had a large IT team of several hundred assorted developers and support staff. Each year, they'd wait for Congress to decide on changes to the tax laws, and then begin the process of implementing those changes in their software so that they could get it to the accountants in time for tax season.

Unfortunately, Congress doesn't take programming time into account when they bicker back and forth over changes to the tax law. Last year, Congress passed a whole slew of changes at the very last minute, leaving insufficient time to implement all the changes in the software (at least not without the usual magic happens here programming of experienced developers). The solution that the blockhead managers came up with was to only implement part of each change prior to distributing the software. They called this "being agile".

When Julie started to prepare tax returns, she realized that the inputs to the same calculation were different on different forms. The numbers were not being propagated to the places that needed them, or worse, were being propagated incorrectly. Additionally, some calculations only performed the first 10 steps of a 12 step form. Upon raising bugs, she was told We know about these issues and have no plans to fix them!

Wait, manually propagating numbers and checking every calculation defeats the purpose of using the software in the first place; the numbers can't be close, they have to be exact and consistent across forms!

The debate about the importance of exact and correct calculations went back a forth and while until it was escalated sufficiently high to warrant an official response:

Our software is only intended to guide you in the general direction of preparing tax returns. It only needs to be reasonably functional, not "useful".

Julie, at this point, was desperate, so she contacted an insider at the company. "Is there *any* chance this is going to be working in time for tax season?"

"Oh, man… noooooo," her contact said. "A lot of the developers are on contract, so somebody ran the numbers comparing the hours of development time against paying penalties from audits. They figured out that they could do about… 90% of the changes. It's cheaper to just pay the penalties later."

Julie is no longer partnering with this conglomerate.

[Advertisement] Application Release Automation for DevOps – integrating with best of breed development tools. Free for teams with up to 5 users. Download and learn more today!

Representative Line: Changing With the Times

The Daily WTF -

Melody got tapped to do a code review on a pull-request from a veteran team-member. It was… an interesting PR, in that very, very little changed. The code was terrible before anyone touched it- for example, the C-file started with 355 lines of variable declarations inside of the main method.

It was, in fact, down around line 354 where Melody noticed the change.

-int right_now_month = 6; // TODO: look up based on current state +int right_now_month = 7; // TODO: look up based on current state

Out of morbid curiosity, Melody went back through the history, and sure enough, once a month, there was a PR that involved incrementing right_now_month.

hljs.initHighlightingOnLoad(); [Advertisement] Incrementally adopt DevOps best practices with BuildMaster, ProGet and Otter, creating a robust, secure, scalable, and reliable DevOps toolchain.

Error'd: The Sky's the Limit!

The Daily WTF -

"Oh boy! I wonder how they'll top this for the end-of-week bonus prize?" writes Zak.

 

"Um,...no? I think?," wrote Peter S.

 

Sean C. wrote, "It would seem that my disks got decrypted over 5 million years ago... Now that's what I call reliable hardware"

 

"I’ve watched the whole series a few times, and I’m pretty sure Scruffy never said anything like this," writes Vincent.

 

Bjorn E. writes, "Started Word, logged into Office 365 and the was greeted with this message instead."

 

"I didn't see any specials that I liked, so I packed up and went to Burger King instead," wrote Rob H.

 

[Advertisement] Release! is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!

The Security Audit

The Daily WTF -

We do our best to anonymize submissions, but there’s always a chance that some dangerously identifying detail slips through. Every once in a while, a submitter contacts us to ask for a modification. More rarely, a submitter’s employer contacts us.

Our rule is to make edits more or less as requested, then move on without comment. There’s nothing about an article so sacrosanct that it’s worth going to war over.

However, we recently got a request that was itself too much of a WTF to pass up.

An old Errord—practically antique—contained a picture of a freaked-out Windows login screen, submitted by someone we’ll call Johnny. The post contained Johnny's name, username, and an Active Directory Domain. It did not identify the company Johnny worked for. But the company—we’ll call them “Lagomorphic Cogitations”—recently performed a security audit and sent us a message:

To: [email protected]
From: [email protected]
Subject: Request for Article Picture Removal (Information Disclosure)

I am reaching out to you about an article posted in 20XX, specifically, the last two screenshots by Johnny in http://thedailywtf.com/articles/someolderrordiwontname.

Gathering an employee’s first name and last name, coupled with the corporate email convention of [First Name].[Last Name]@LagomorphicCogitations.com may allow attackers to gather a list of possible targets for phishing attacks. If the article can be edited to remove the last section from Johnny it would be greatly appreciated as we are looking to limit our exposure of cooperate information. Thank you!

-Hunter Jacobs

We did some basic due diligence, confirming the email looked authentic and that someone with this name worked on Lagomorphic Cogitations' security team. I also happen to know Johnny, and confirmed with him that he'd left that job over five years ago, and yes, that particular submission came in during his tenure there. We removed the offending image. Frankly, we didn’t do much else. It’s a minor edit to an old Errord, and we’re not that concerned.

There’s a fair bit to unpack here, though. After all, our article made no mention of Lagmorphic Cogitations. To find it, they must have been searching employee names—not just current employees, but also ex-employees. (A quick check on LinkedIn shows me 45,000 people who work or have worked for Lagomorphic Cogitations.)

After that incredible slog, someone then had to click on the article, look at the image, recognize the Lagomorphic Cogitations domain, then contact us directly to request a removal. Even with modern technology, there’s only a small degree of automation that could've been involved in this process.

If nothing else, this gives us a sense of how much effort Lagomorphic Cogitations puts into these audits. Unfortunately, their attempt to prevent information disclosure has led to, well, more of it, hasn’t it? Before we received their email, we didn’t know Johnny had ever worked for them. We didn’t know that screenshot was their logon. Heck, we didn’t know their email address pattern, even if [First Name].[Last Name]@LagomorphicCogitations.com is pretty easy to guess.

Now we know all of those things.

Based on my own personal experience in these kinds of environments, I have a hunch about what happened. Lagomorphic Cogitations almost certainly didn’t perform this audit themselves; this was an outside contractor thing. They handed the contractor a check, and a few months later, the contractor came back with a spreadsheet listing “potential security risks.” The security company looks better the more they find, and they’re also not being paid to do more than provide a high-level triage, so this spreadsheet contains anything and everything they could possibly dig up.

From there, the executive who commissioned that study handed the spreadsheet to a pool of middle managers and said, “Deal with these.” The spreadsheet filtered down the hierarchy until it landed on the desk of poor Hunter, who just followed the policy he was given: Tell them to take it down because it’s an information disclosure. If they don’t, escalate.

After we took down the image, Hunter changed the background of the row in his spreadsheet to green. Then he moved on to asking a Facebook group to remove a picture of a cute bunny, because it happens to be owned by an old employee and its name might be part of their password.

At the end of the quarter, Hunter’s boss will brag to the executive board about how his team has resolved 3,421 security issues identified by the audit. The board will nod sagely and pat themselves on the back for writing that check.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!

CodeSOD: The Story of Things

The Daily WTF -

Every line of code tells a story. It never just… appears. Someone made and crafted that code. There’s a story, and an explanation for how that code could be. The world, even the bad, awful corners of it, makes sense and can be understood.

For example, Luke sends us this block.

/** * Format a string as a phone number mm/dd/yyyy */ public static String formatPhone(String phone) { String s = ""; if (phone != null) { s = phone.trim(); } return s; }

I imagine, the thinking went like this: “I need to write a function to format a phone number, so I’ll copy the code for the function to format dates, and I’ll replace every instance of date with phone. Oh, wait, the comment is unclear. I’ll make that phone number”

Similarly, this Anonymous PHP code tells its own story:

if(!$this->auth_model->in_group('Manager')) { $data->userLevel = 1; } else { $data->userLevel = 3; } if($data->userLevel == 3) { redirect('index', 'refresh'); //In CodeIgniter, this triggers a 301 redirect to another page. }

The story, I imagine here, was a case of conflicting terminology. Once upon a time, someone came up with the idea of “User Levels”. Someone else thought that was needlessly cryptic, and wanted to give them names. Then this developer got confused, and decided to translate everything back into user levels. This particular if block appeared dozens of times in the code, and it was the only place that userLevel was eveer used. Our anoymous submitter ripped them out.

Finally, a different Anonymous contributor found this block:

def formatDate(date): date = time.strptime(str(date), '%Y-%m-%d') date = str(date.tm_year)+'-'+str(date.tm_mon)+'-'+str(date.tm_mday)

Now, the developer responsible for this knew that they needed a date in a Year-Month-Day(%Y-%m-%d) format. So they started out by taking a date object, and converting it to a string, using str(date). This converts it into a %Y-%m-%d format. They then parse that into a date object- specifically a struct_time, using time.strptime. Once they had that, they could then use the various properties of the struct_time to build a %Y-%m-%d format.

I take back what I said. The world is a chaotic, disordered place, effects may proceed without cause. Sense is nonsense, and paradoxes are the only form of truth. There’s no explanation for this code, or for… anything, is there?

hljs.initHighlightingOnLoad(); [Advertisement] Release! is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!

Time To Transfer

The Daily WTF -

When people think about government, they usually think about a President or Prime Minister, Senators, MPs, or what have you. But government isn't just a handful of people at the top of the food chain: there's government all the way down to the city level, quietly making the country run. Driver's licenses have to be issued, as do pet licenses. Buildings have to be inspected and certified. All those elevator certificates get printed up somewhere. Increasingly, these small functions are being computerized—in bits and pieces, in incompatible systems—and hooked up to the Internet.

Lisa was the lead engineer for one of these public websites. At its core, it took in personally identifying details and spat out some sort of official document. This meant they had to deal with the PII issues that come with taking people's information: encrypting and salting the data, securing the database backend, et cetera.

One of the pieces in this chain was a separation of data: until the user had paid for the document, proving their identity (or at least their possession of the credit card for the person they claimed to be), their data sat in a frontend database accessible to the Internet. After payment was taken, the data was sent to a more secure database in the backend and removed from the potentially hackable frontend. The frontend ran in a VM that could only make an outgoing connection to the database. It could receive incoming connections and respond, but not initiate them. Basic security for this type of system.

There was one issue, however, that Lisa struggled to track down. It seemed that a small percentage of users, fewer than 1%, were getting an error page immediately after payment. Their application was fine; payment was received, and their document was sent to them along with a confirmation. But they saw an error page suggesting they hadn't completed their transaction.

When Lisa managed to catch the issue in the act, she was able to reconstruct the sequence from the logs:

  • The user entered their card info
  • The payment processor accepted the payment
  • The application marked the record as paid
  • The application responded with the redirection back to the confirmation page
  • The transfer service kicked in and moved the data
  • The user landed on the confirmation page—and the record was no longer there to display.

In other words, a timing issue. So far, this was just a run-of-the-mill everyday problem. Race conditions happen all the time, after all. The problem was, Lisa had already fixed this race condition: the logic indicated that the system would wait at least 5 minutes before moving the data from the frontend to the backend, to allow for the confirmation page to be generated.

So, what gives? Lisa wondered.

The logic was still in place. In fact, the logs showed that the data hadn't been moved until 5 minutes after it was marked to be moved. But the confirmation page had generated in mere seconds. How could this possibly have occurred?

"It just doesn't make sense," she complained to her coworker.

"What time is it on the server right now?" he asked with a frown.

And that was it: the transfer service was 6 minutes off from the frontend box. As soon as it was marked eligible for transfer, the backend box would move the data. If the periodic service ran right when the row was generated, the user would get an error.

With a groan, Lisa put in a ticket to have both boxes sync to a time server.

[Advertisement] Release! is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!

Chapter Three: How to Prevent Duplicate Terms During a Drupal 8 Migration

Drupal Planet -

In this post I will show a custom process plugin that I created to migrate taxonomy terms. The plugin handles the creation of new terms and prevents duplicates.

Below is a portion of the migration template. In the example, I am migrating new terms into keywords vocabulary via field_keywords field.

field_keywords: - plugin: existing_term # Destination (Drupal) vocabulary name vocabulary: keywords # Source query should return term name source: term_name - plugin: skip_on_empty method: row

This is the source code for the process plugin.

Pages

Subscribe to Heydon Consulting aggregator