RWAPP

My name's Rob. I'm an iOS software development engineer for Capital One UK. I talk & write about mobile accessibility.

JSON Deocding in Swift 4

You’d be forgiven for thinking iOS 11 is only about augmented reality, so for anyone yet to have a use for AR in their apps, perhaps there’s not a ton of work this summer? But there is one change for iOS 11 that I’ve been looking forward to adding to my apps - JSON decoding thats built in to Swift 4. To date I’ve been making use of the great AlamoFire and SwiftyJSON libraries to process this for me. But being able to replace any 3rd party code for your own is always preferable for reasons of maintenance, portability, security, etc etc etc. Plus, JSONDecoder has reduced the lines of code I’m using, and has added safety.

Here I’m loading JSON from a remote service, and JSONDecoder is returning this to me as a Swift tuple. Its also possible to encode JSON using almost the same process in reverse, with JSONEncoder, but I’m not going to touch on that here.

Defining a Model

The first step is to define the model for the JSON you’re receiving, this is done in the form of a struct that conforms to Codable. Lets imagine we want to parse the following JSON:

1
2
3
4
{"name":"Wellard",
"colour":"Black",
"age":4,
"breed_name":"Belgian Tervuren"}

The struct we’d write would look something like this

1
2
3
4
5
6
struct Dog: Codable {
var name: String
var colour: String
var age: Int
var breed_name: String
}

Remapping Variable Names

Great, except JSON often has a couple of quirks that doesn’t always map perfectly to swift. JSON regularly uses snake_case to name variables, which doesn’t match nicely with Swift’s camelCase. So we’ll want to rename some. A CodingKeys enum allows us to do this. Somewhat irritatingly, we must include all keys in this, even ones we’re not renaming, so this could become laborious for larger JSON objects. Possibly something Sourcery could help you out with?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
struct Dog: Codable {

enum CodingKeys : String, CodingKey {
case name
case colour
case age
case breedName = "breed_name"
}

var name: String
var colour: String
var age: Int
var breedName: String
}

Notice above we use the JSON name in the enum, and those we want to rename we give a string for the new variable name. Then in the body of the struct, use the new Swift variable name. Its also worth mentioning here, that you can also ignore values. If your JSON returns something you have no need for, just don’t add the variable name to your struct, and Swift will discard it.

Wrappers and Nested Elements

JSON often features wrappers, like returning objects inside a ‘results’ object. These can be processed by nesting it inside another struct. What if we’re returned an array of dogs inside a ‘kennel’ element, the JSON might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
{"kennel":[

{"name":"Wellard",
"colour":"Black",
"age":4,
"breed_name":"Belgian Tervuren"},

{"name":"Bouncer",
"colour":"Golden",
"age":7,
"breed_name":"Labrador Retriever"}

]}

To decode this we need to define a struct that contains an array of the Dog structs we created earlier.

1
2
3
4
5
struct Kennel: Codable {

var kennel: [Dog]

}

The approach is similar if our JSON contains nested elements. Lets say we want to list each dog’s favourite toy, each toy has values like type and colour, then we’d define a third struct like so:

1
2
3
4
5
6
7
struct Toy: Codable {

var name: String
var colour: String
var type: String

}

Then in the Dogs struct we’d add a variable for toy, with the type of Toy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
struct Dog: Codable {

enum CodingKeys : String, CodingKey {
case name
case colour
case age
case breedName = "breed_name"
case favouriteToy = "favourite_toy"
}

var name: String
var colour: String
var age: Int
var breedName: String
var favouriteToy: Toy
}

Decoding

Once we’ve defined our model to match the JSON input, decoding it into Swift is essentially trivial. We use Swift 4’s new JSONDecoder and the decode method. This takes two arguments, firstly the model. As our JSON is wrapped in a Kennel value we use Kennel.self. Then we pass the data we received from our URLSession. Wrap this in a try? to catch if the JSON you’re feeding in doesn’t match your model.

1
2
3
4
if let decoded = try? JSONDecoder().decode(Kennel.self,
from: data!) {
print(decoded)
}

Done.

To bring this all together, here’s the code we’ve created so far. Lets assume the JSON data we’re provided is…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{"kennel":[

{"name":"Wellard",
"colour":"Black",
"age":4,
"breed_name":"Belgian Tervuren",
"favourite_toy":{

"name":"Squeaky",
"colour":"Yellow",
"type":"Ball"}

},

{"name":"Bouncer",
"colour":"Golden",
"age":7,
"breed_name":"Labrador Retriever",
"favourite_toy":{

"name":"Mr Heffalump",
"colour":"Blue",
"type":"Stuffed Elephant"}

}]}

Then our Swift code will look like…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
struct Dog: Codable {

enum CodingKeys : String, CodingKey {
case name
case colour
case age
case breedName = "breed_name"
case favouriteToy = "favourite_toy"
}

var name: String
var colour: String
var age: Int
var breedName: String
var favouriteToy: Toy
}

struct Toy: Codable {

var name: String
var colour: String
var type: String

}

struct Kennel: Codable {

var kennel: [Dog]

}

let sessionConfig = URLSessionConfiguration.default
let urlSession = URLSession(configuration: sessionConfig)
let url = URL.init(string: "http://supercooldogjsondatabase.com")

let dataTask = urlSession.dataTask(with: url!) { data, response, error in

if error != nil {
print(error?.localizedDescription ?? "")
} else {
if let data = data {
if let decoded = try? JSONDecoder().decode(Kennel.self,
from: data) {
print(decoded)
}
}
}
}

dataTask.resume()

Where Next

Try this out on some actual JSON to get used to using it, MAAS is one of the JSON APIs I use in Mars Watch to get weather reports from Mars. Its a pretty simple set of data to use to try out JSONDecoder, and one that I used all the techniques above to get the best results for me.

Tutorial - Vision and Core ML - Live Object Detection in iOS 11

While WWDC 2017 may have not been one of the most announcement-packed WWDC’s ever, there’s no doubt over the features Apple is most proud of and sees the biggest opportunities for the future. Augmented Reality using ARKit, Machine Learning with Core ML, and pairing these with Vision and Metal for AR and VR. My initial thought, I’m sure the same as many other viewers, was that this technology is for would be reserved for the experts, and fans of really hard sums. But Apple’s implementation is mind-blowingly simple, and can really be added into almost any app in just a few minutes.

In this tutorial we’ll build Core ML and Vision Framework app for detecting objects in real time using your phone’s camera, similar to the one demoed in WWDC session 506. To begin you’ll need the Xcode 9 Beta, and you’ll need the iOS 11 Beta installed on your testing iPhone or iPad, meaning for now this tutorial is just for registered Apple Developers. But if you’re not one right now, this will all be freely available in the autumn. If you are registered, head over to Apple’s Developer Downloads page, and grab what you need. Bare in mind that this tutorial is written against iOS 11 Beta 2, so things could change a little by the final release.

This tutorial assumes some basic iOS Swift programming knowledge already, but I’ve tried to keep it as beginner friendly as possible. Please drop me an email to rw@rwapp.co.uk if there’s anything unclear.

Setting up the project

For this demo we’re going to do the object detection live, showing the camera, and overlay text on the screen telling us what Core ML can see. To begin, create a new project in Xcode 9 Beta and choose a Single View Application. Call your project Object Detector. Head to ViewController.swift and add a couple of @IBOutlets. One for a UIView we’re calling cameraView - this will be your ‘viewfinder’. Add a second @IBOutlet for a UILabel called textLayer - this will be our output of what Core ML has detected.

Open the Main Storyboard and drag on a blank view. Resize it to fit the screen and pin it to the edges. Now drag a label on to the view. Make sure this is topmost on the root view, not on top of the view you just added, or it will be obscured by the camera output. Position the label to the top layout guides and fill the guides left & right. Delete the text, set the text colour to something bright so you’ll see it over the image, and change the label to be 6 lines long. Finally add some suitable constraints to the label. Join up the IBOutlets you made earlier to the view and label.

To use the camera, we need to ask permission first, so head to info.plist and add a key for ‘Privacy - Camera Usage Description’ the string value can be anything you like for now, this is the message shown when asking permission for camera use at first launch.

Displaying the camera output

Now, to the code. First we need to use Vision to create an AVCApture session, this provides us with the live camera view, and also feeds the data from the camera into Core ML. Import AVFoundation and Vision. Head back to ViewController.swift. We need to set up a video capture delegate, so add AVCaptureVideoDataOutputSampleBufferDelegate to your class definition. Add the following code between your IBOutlets and viewDidLoad.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// set up the handler for the captured images
private let visionSequenceHandler = VNSequenceRequestHandler()

// set up the camera preview layer
private lazy var cameraLayer: AVCaptureVideoPreviewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)

// set up the capture session
private lazy var captureSession: AVCaptureSession = {
let session = AVCaptureSession()
session.sessionPreset = AVCaptureSession.Preset.photo


guard
// set up the rear camera as the device to capture images from
let backCamera = AVCaptureDevice.default(.builtInWideAngleCamera,
for: .video,
position: .back),
let input = try? AVCaptureDeviceInput(device: backCamera)
else {
print("no camera is available.");
return session

}
// add the rear camera as the capture device
session.addInput(input)
return session
}()

The code above sets up the camera preview layer, then defines the capture session, telling iOS which camera to use, and what quality we’re looking for.

Now we’ve set that up, we need to add the camera preview layer to the display, and tell iOS to start capturing from the camera. Add the following code to viewDidLoad.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// add the camera preview
self.cameraView?.layer.addSublayer(self.cameraLayer)

// set up the delegate to handle the images to be fed to Core ML
let videoOutput = AVCaptureVideoDataOutput()

// we want to process the image buffer and ML off the main thread
videoOutput.setSampleBufferDelegate(self,
queue: DispatchQueue(label: "DispatchQueue"))

self.captureSession.addOutput(videoOutput)

// make the camera output fill the screen
cameraLayer.videoGravity = AVLayerVideoGravity.resizeAspectFill

// begin the session
self.captureSession.startRunning()

Then to make sure the viewfinder fills the screen, add a viewDidLayoutSubviews override.

1
2
3
4
5
6
override func viewDidLayoutSubviews() {
super.viewDidLayoutSubviews()

// make sure the layer is the correct size
self.cameraLayer.frame = (self.cameraView?.frame)!
}

Set your target to run on your testing device (as this uses the camera, you’ll just get a blank screen on the simulator) and hit Build and Run. After granting access to the camera, you’ll be greeted with a full screen output of your devices rear camera view.

Missing something? Get the code up to this point on Github.

Adding Core ML Magic

First of all, we need a Machine Learning Model. Fortunately that’s a simple as downloading one from Apple’s website. Head over there now and download the Inception v3 model. Add this file to your project.

We now need to get the data from the AVCaptureSession, and feed that our Inception v3 Model. We do this using a AVCaptureVideoDataOutputSampleBufferDelegate method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {

// Get the pixel buffer from the capture session
guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

// load the Core ML model
guard let visionModel:VNCoreMLModel = try? VNCoreMLModel(for: Inceptionv3().model) else { return }

// set up the classification request
let classificationRequest = VNCoreMLRequest(model: visionModel,
completionHandler: handleClassification)

// automatically resize the image from the pixel buffer to fit what the model needs
classificationRequest.imageCropAndScaleOption = VNImageCropAndScaleOptionCenterCrop

// perform the machine learning classification
do {
try self.visionSequenceHandler.perform([classificationRequest], on: pixelBuffer)
} catch {
print("Throws: \(error)")
}
}
}

Just before we handle the results Core ML gives us, two functions to help us update the UI, and to map the Core ML result into something more meaningful

1
2
3
4
5
6
7
8
9
10
11
12
13
14
func updateClassificationLabel(labelString: String) {
// We processed the capture session and Core ML off the main thread, so the completion handler was called onthe the same thread
// So we need to remember to get the main thread again to update the UI

DispatchQueue.main.async {
self.textLayer?.text = labelString
}
}

func textForClassification(classification: VNClassificationObservation) -> String {
// Mapping the results from the VNClassificationObservation to a human readable string
let pc = Int(classification.confidence * 100)
return "\(classification.identifier)\nConfidence: \(pc)%"
}

Finally, to handle the Core ML results themselves. Here we want to make a few quick checks to make sure everything is as we expect, then we need to filter to get the accuracy and data that we want. For this example we just want to display all the objects we’ve found, so the only filtering we’ve done is to cut off the last 20% of results. In the real world, you’ll probably want to look for specific objects, so use .filter here to narrow those down. More on that below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func handleClassification(request: VNRequest,
error: Error?) {
guard let observations = request.results else {

// Nothing has been returned so we want to clear the label.
updateClassificationLabel(labelString: "")

return
}

let classifications = observations[0...3] //taking just the top 3, ignoring the rest
.flatMap({ $0 as? VNClassificationObservation}) // discard any erroneous results
.filter({ $0.confidence > 0.2 }) // discard anything with less than 20% accuracy.
.map(self.textForClassification) // get the text to display
// Filter further here if you're looking for specific objects

if (classifications.count > 0) {
// update the label to display what we found
updateClassificationLabel(labelString: "\(classifications.joined(separator: "\n"))")
} else {
// nothing matches our criteria, so clear the label
updateClassificationLabel(labelString: "")
}
}

We only filtered through the top 3 results provided by the Machine Learning model. For this project we just want to know the most probable results. The observation results are provided in decreasing order of certainty, so we know anything else is likely to be a false positive. Click on the model in the sidebar, and you’ll see in the model description Apple lists a ‘top-5 error’ rate. This is the percentage of false positives provided for the model in the top 5 results when tested. In real world situations using a live camera, this is likely to be higher.

Description: Detects the dominant objects present in an image
from a set of 1000 categories such as trees, animals, food, vehicles,
person etc. The top-5 error from the original publication is 5.6%.

Iterating through all the provided guesses from the model is going to use mote resources, and is unlikely to yield any more real positives. If you’re looking of specific items and you’re doing your own filtering, you may find better end results by looking at more of the model’s output.

Doggo

Where to go from here

If you are looking for specific objects, I’d suggest cutting the confidence level right down to 10% or even less. Then add an extra filter step with your own custom logic to narrow down what you’re looking for.
I found the Inception v3 model to sometimes be a little too specific at times, and sometimes not specific enough. It all depends on the category of object you’re looking for, and wether the model has already been trained on that category. Its worth checking out other models to make sure you get the one that works best for your project. Apple provide some popular models ready to drop in to your project. Just change the model name in the line guard let visionModel:VNCoreMLModel = try? VNCoreMLModel(for: Inceptionv3().model) else { return } to match the name of the model you choose. If you find another model that does the job, Apple provide Core ML Tools to convert machine learning models to the Core ML format.

I hope you found this quick introduction to Core ML and Vision helpful, if you’ve got anything to add, or any questions, please let me know by email rw@rwapp.co.uk, and I’d love to see any apps you make using these frameworks, so please drop me a link when its on the App Store.

Full tutorial code available on Github

Derby Beer Festival Apps 2017

Apps for the 2017 Derby Beer Festival are now available from the App Store and Google Play.

I’ve listened to a lot of feedback for these versions, so I really hope you enjoy Ising them. There are stability and reliability improvements and more timely updates. There’s a new, more positive, recommendations system. Its easier to find drinks that have run out, and easier to find drinks that are gluten-free, vegan, and more.
I’m always keen to improve these apps, so please drop me an email to rw@rwapp.co.uk if you have any feedback.

The 40th Derby CAMRA City Charter Summer Beer Festival will be back from the 5th to the 9th July 2017 on the Market Place in the center of Derby.

Channel 4's Loaded

Back in the autumn I helped my good friend and talented TV and Film production designer Matt Clark with some tips on what screens should look like for mobile app development. You can now see the results of that advice on Channel 4’s Loaded.

In return Matt hid references to RWAPP in the show. See if you can spot them!

Loaded screen with RWAPP laptop sticker

Check out Matt’s website, FFO Dr Who and Red Dwarf.

Mars Watch

Today my newest iOS App is available to download for free from the App Store.

Mars Watch is the perfect companion for anyone who is planning a trip to the Red Planet.

Get the latest local time on Mars, along with recent weather reports, so you know which space suit to pack. Includes an iMessage App, Apple Watch App and Today Widget.

Support
Credits

Download for free on the App Store

Mars Watch Credits

Thanks for using Mars Watch.
If you’re looking for support, please check out the support page

I am indebted to the great work done by these people.

Mars time calculations based on work by James Tauber and NASA.

Mars weather data comes from NASA’s Curiosity rover on Mars and the REMS.

Mars rover photos from NASA.

Libraries

Alamofire
Font Awesome
Font Awesome Swift

Thanks to Mike.

Mars Watch © 2017 RWAPP Rob Whitaker

If you enjoy the free Mars Watch app, please consider sending me some beer money.

Mars Watch Support

Thanks for using Mars Watch. I hope its been an invaluable tool to help you plan your next trip to Mars. If there’s anything in the App you’re not sure of, hopefully you’ll find an answer here.

The time on my Apple Watch Complication isn’t updating

To save power, your Apple Watch only updates your complications when its most efficient to do it. That means I can’t rely on the time updating as often as I’d like. I’ll keep working on getting this as accurate as possible, but it seems Apple never expected time other than earth times to be displayed!
To make sure you get the most accurate time, tap the complication to load the Apple Watch App and get the current time.

The weather data hasn’t changed for a while

NASA’s Curiosity rover sends data back periodically, on average around every 48 hours, but sometimes it can be longer. Try waiting a day or two for an update.

I’d like to see temperatures in Fahrenheit

You’re wrong. You’d like to learn to use a practical, non-antiquated, logical unit of measurement.

Got a question not listed here?

Please feel free to email me, rw@rwapp.co.uk

If you enjoy the free Mars Watch app, please consider sending me some beer money.

Eat Me Support

Thanks for using Eat Me. I hope its been an invaluable tool to help you manage your kitchen. If there’s anything in the App you’re not sure of, hopefully you’ll find an answer here.

How do I share my list with someone else in my family/between devices?

This isn’t yet possible, but I hope to add this feature soon. Thanks for asking, if you’d really love this feature, why not email me to to tell me how much you’d appreciate it.

How do I remove ads?

I hope to add this feature soon. You know, thinking about it, if every person who liked the app dropped just a little money my way, I probably wouldn’t need to put adds in the app at all.

Got a question not listed here?

Please feel free to email me, rw@rwapp.co.uk

If you enjoy the free Eat Me app, please consider sending me some beer money.

What I Learned This Week 17/11/16

What I Learned This Week is a weekly blog of small dollops of info I remember from the previous 7 days. It covers a wide range of topics, often tech, but sometimes wrestling, beer and a random selection of other things too. Have anything to add, find me on Twitter

History Isn’t Comfortable

Historical reenactments are a popular pastime in the southern USA, the former confederate states re-play their somewhat questionable history in a way that skin-deep looks like harmless fun. The moment you begin to question what is being celebrated it raises some awkward questions about where the US still is regarding race.
I found this Guardian Long Read (also as a podcast) about a rather different kind of reenactment lead me to a lot more questions, and no answers. The whole event is a fascinating investigation of race, history and society in a way that feels very uncomfortable. As to wether this is as distasteful as celebrating the confederate flag, I think that could be easily argued. Ultimately though it is an important piece of folk history that I think many would happily forget.

The Internet Has A Human Firewall

The internet, for all its brilliance, can be a horrible place. For most of us its always possible to move past the worst parts, they can be avoided, or prevented, or we can just turn off. Imagine not just being subjected to the internet’s detritus, but to have it directed at you, all day, every day.
Moderation is difficult, so much of it has now been removed from automated systems, and into the hands of real people. Perhaps with the advancement of Neural Networks the task will get easier so we don’t have to have a firewall of real people so the rest of us don’t get upset. It would sure be a more worth use than Pictionary.

Frogs Don’t Like Space

I wanted something lighter to end on, but I’m not sure if this is just as grim. Frogs don’t like space.