Unlock the Editor’s Digest free of charge
Roula Khalaf, Editor of the FT, selects her favorite tales on this weekly e-newsletter.
One month in the past, we acquired (by way of freedom of data request) the Workplace for Nationwide Statistics’ steerage on what its workers and brokers ought to be searching for when observing the “basket” of things used to watch inflation.
We wrote then:
We’ll attempt to make a tidier model of the ONS’s desk quickly
Within the meantime, stuff has occurred. However now we’re SO again, and we’ve received the mega spreadsheet to show it.
Should you’re from a hedge fund and are simply right here to scrape stated spreadsheet, please scroll down. Should you’re taken with a woeful story, persist with us.
A woeful story, pt.1
We talked about final month that:
As a result of civil servants are merciless, the doc’s formatting is horrible, half of its textual content isn’t precise textual content, and there are redactions. 🙃
Wah, wah, wah — sure, journalism actually IS laborious and don’t let anybody inform you in any other case.
The ONS’s response to our freedom of data request landed as a 70-page PDF file, which might be downloaded right here.
The PDF is, principally, a spreadsheet, consisting of merchandise names, and as much as six “help line” columns detailing precisely what the ONS considers legitimate for that class.
So, as an illustration:
That is clearly odd. Why a steady string of textual content is being written throughout six columns on this manner, we don’t know.
In fact, not all the gadgets are this oddly separated, eg:
Whereas others are far more condensed, equivalent to:
There is no such thing as a obvious rhyme nor purpose to why the steerage is usually unfold out throughout columns, neither is there something resembling a constant fashion to how issues are written. It very a lot feels just like the product of years of advert hoc tweaks and edits (presumably in response to requests for clarification), and provides us concrete poetry flashbacks.
Readers might have observed references to codes equivalent to N and C in these assist traces above. N means a brand new product is non-comparable to a beforehand noticed product, whereas C means it’s. Easy!
A woeful story, pt.2
We’re not precisely certain how the ONS managed this, however the PDF’s pagination per merchandise loosely goes like this:
First pages (labelled 1–35, odd numbers in PDF)
Merchandise description / Assist line 1 / Assist Line 2
Second pages (labelled 36–70, even numbers in PDF)
Assist line 3 / Assist Line 4 / Assist line 5 / Assist line 6
Clearly this isn’t a giant problem in and of itself, however we wished to transform this right into a extra usable format and right here (non-exhaustively), are a number of the different points the PDF has:
— Some pages, together with the primary, haven’t been saved as textual content
— Some cells are filled with random line breaks and different typographic artefacts
— Nothing redacted ever pastes good
— Textual content copied from cells got here out as unbroken strings when pasted
— In Acrobat and OS X preview, some pages copied sideways, natch.
— Lom
— Lom
— Lom
We reached out to the ONS and requested if we may merely get the underlying spreadsheet, and within the meantime went to work doing issues semi-manually (copying particular person cells or typing them out by hand).
A woeful story, pt.3
Though [REDACTED] was truly extraordinarily useful — and there’s a twisted logic to how issues had been accomplished right here — so it was that we ended up primarily deriving our data from the optical-character-recognition generated .docx model of a PDF model of an Excel spreadsheet.
Behold, a basket
A while later, we’d received every thing right into a single spreadsheet. Caveats:
— we have now undoubtedly made some typos given the quantity of handbook copying of weirdly-written information we needed to do
— we’ve recreated the steerage within the precise fashion it was written, which is usually massively inconsistent.
— in some cases, the OCR course of utilized by the ONS signifies that letters with descenders that seems on the backside of a web page have scanned weirdly, or that some letters are borked (eg we needed to repair “fluten” to “gluten”)
As talked about, the column divisions made little or no total sense. So we concatenated them to make every set of steerage learn as one paragraph.
For everybody else, take pleasure in (nb there’s a direct obtain hyperlink within the backside proper in order for you the uncooked information):
Statement notes on statement notes
Final time round, we talked about a number of the oddities on this doc. The laborious strategy of making ready this spreadsheet brought about us to notice a number of extra:
— a toddler’s delicate toy/teddy bear CANNOT be a hand puppet
— a toddler’s child doll will need to have a plastic component
— gold rings aren’t allowed to be rose gold (don’t be fundamental)
— natural bread ought to solely be priced if it’s “representative” of the store it’s from, no matter meaning
— quiche crusts are non-compulsory
— observers should specify whether or not popcorn is nice/wage/each, nevertheless it doesn’t truly matter
— the steerage on livery fees is… thorough
— 2-in-1 bathe gel merchandise aren’t allowed, sorry Boris
— hen nuggets don’t depend as hen for the needs of a hen and chips takeaway
Specifically, we discovered ourselves drawn to quite a lot of gadgets the place there may be little or no assist supplied, together with…
— Digital Media Participant
— Pc Software program
— Cellular Telephone Purposes
— Interchangeable Lens Digital Digicam
— Laptop computer Computer systems
…and dozens extra, all of which have massively variable product sorts and prices. ¯_(ツ)_/¯
In reality, nearly every thing in right here appears absurd for those who look carefully sufficient. Oh effectively.
Additional studying:
— small caged mammal (FTAV)
— small caged mammal infinity (FTAV)
— small caged mammal revelations (FTAV)
— small caged mammal merchandise (Redbubble)
