Ohhh man this is getting close now.
I am starting to really want to put this project to bed, and I think I will get there soon.
No time to waste so let’s recap.
If you haven’t followed along, and want to know what this project is, start here.
Where did I start
At the start of the week I had a fairly bug free IOS client, connected to an increasingly stable server and db, with a relatively solid set of instructions for creating, updating and deleting a set of notes related to a specific user.
Authentication was still working nicely, but my app shat itself as soon as it lost internet connection.
The focus of this week was:
-
Getting IOS app into the App Store
-
Offline mode
-
Client for desktop
How’d I do?
I added createNote and noteCreated actions to my notes protocol, in order to make things more explicit, which worked nicely.
Offline mode
I came up with an initial solution which appears to work pretty well. I may change this going forward (see more below), but this is at least a start.
I keep track of the offline/online status in the IOS client, and at the point a user does a note action (create, update, delete), if they are online push the update straight to the server, or if they are offline, update the locally stored copy of the notes, store the action locally on the device, and then when the device comes back online, process all the queued up actions.
This looks like this for the update action:
NotesService.swift:
public func updateNote(id: String, title: String, body: String, prevNote: Note, context: NSManagedObjectContext) {
// Figure out any diffs
let titleDiff = NotesDiffer.shared.diff(notes1: prevNote.title!, notes2: title)
let bodyDiff = NotesDiffer.shared.diff(notes1: prevNote.body!, notes2: body)
let payload: [String: Any] = [
"id": id,
"title": titleDiff,
"body": bodyDiff,
]
if (self.online) {
// we are online, push action straight to server
self.socket?.emit("updateNote", payload)
} else {
// no internet :(
// find the existing note stored in CoreData locally
let note = Note.noteById(id: id, in: context)
// update existing local copy of note
Note.updateTitle(note: note!, title: title, in: context)
Note.updateBody(note: note!, body: body, in: context)
// delegate update responsibility to OfflineChanges service
OfflineChanges.updateNote(payload: payload)
}
}
OfflineChanges.swift
private static let key: String = "offlineUpdates"
private static let defaults = UserDefaults.standard
public static func updateNote(payload: Any) {
var offlineUpdates = defaults.array(forKey: key)
// put action and payload in an array
let action = ["updateNote", payload]
if (offlineUpdates != nil) {
offlineUpdates!.append(action)
} else {
offlineUpdates = [action]
}
// store updated offline updates to user defaults
defaults.set(offlineUpdates, forKey: key)
}
// loop through all stored offline updates, and push them up to server
public static func processOfflineUpdates(socket: SocketIOClient?, done: @escaping () -> Void) {
let offlineUpdates: [[Any]]? = defaults.array(forKey: key) as? [[Any]]
if (offlineUpdates != nil && offlineUpdates?.count ?? 0 > 0) {
socket?.emit("offlineUpdates", offlineUpdates!)
socket?.once("offlineUpdatesProcessed") { data, ack in
done()
}
} else {
done()
}
defaults.set([], forKey: key)
}
Then, back in the NotesService, when we reconnect, after authentication, process all the stored updates:
self.socket?.once("authenticated", callback: { _, _ in
OfflineChanges.processOfflineUpdates(socket: self.socket) {
self.socket?.emit("getInitialNotes")
}
self.socket?.once("initialNotes") {data, ack in
let stringifiedJson = data[0] as? String
if (stringifiedJson != nil) {
self._onInitialNotes!(NotesToJsonService.jsonToNotesDictionary(jsonString: stringifiedJson!))
} else {
self._onInitialNotes!([:])
}
}
});
I’m overall happy with this approach.
The one thing I think I might end up changing is the explicit online/offline detection
I think it might be more reliable to instead check that the server received the action within a set amount of time.
If it doesn’t respond with a ‘yes I got that message’, assume we are offline and queue up the action for later as detailed above.
Let’s distribute this thing! (to a tiny set of initial users)
Now that I had a client working to a level I was happy with, it was time to get it in front of people.
I dutifully signed up to Apple’s developer program, paid my fee and carried out the steps to push one of my builds to the ‘App Store connect’ dashboard.
It was quite a nice process, which after setup could be managed from within Xcode.
Apple offers a beta testing product called ‘TestFlight’, which allows you to send email invitations to people, allowing them to install your app via the ‘TestFlight’ app.
I was able to convince five people to install the app and report any issues they found.
So far, no major issues, but I don’t think that means it is bug free alas.
Based on this extremely limited testing, I’m now pretty happy with pushing to actually submit something to the App Store, and that will be the focus of next week.
Desktop
A large part of why I’m making this app is that it is something I want to use.
In order for me to actually find it useful, it needs to have a desktop client, at least on Mac.
After some brief tinkering with native MacOS tooling, I once again said “fuck it I’ll just do Electron”.
I started a new project, and pulled in the parts from my initial Electron prototype that were good.
A benefit of spending so long with the IOS client finessing the notes protocol and online/offline functionality is that it made implementing the Electron client something of a dream.
I already had the Auth stuff done, so it was a case of implementing the new master/detail views for handling multiple notes (the previous Electron app only supported one page of notes per user), coming up with a way of storing notes locally to the user’s machine, and implementing the same set of actions as on the IOS client.
Desktop local storage
I used sqlite, via the sqlite3 npm package, and put together a service for handling CRUD operations:
note-storage.service.js
const { app } = require("electron");
const path = require("path");
var sqlite3 = require("sqlite3").verbose();
const db = new sqlite3.Database(path.join(app.getPath("userData"), "notes"));
db.serialize(() => {
db.run(`
CREATE TABLE IF NOT EXISTS notes (
id TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
body TEXT NOT NULL
)`);
});
app.on("quit", () => {
db.close();
});
const getNotes = (done) => {
db.serialize(() => {
db.all(
`
SELECT * FROM notes
`,
(err, results) => done(err, results)
);
});
};
const getNoteById = (id, done) => {
db.serialize(() => {
db.get(
`
SELECT * FROM notes
WHERE id="${id}"
`,
(err, result) => done(err, result)
);
});
};
const createNote = (note) => {
db.serialize(() => {
db.run(`
INSERT INTO notes (id, title, body)
VALUES("${note.id}", "${note.title}", "${note.body}")
`);
});
};
const updateNote = (note) => {
db.serialize(() => {
db.run(`
UPDATE notes
SET title="${note.title}",
body="${note.body}"
WHERE id="${note.id}"
`);
});
};
const deleteNote = (id) => {
db.serialize(() => {
db.run(`
DELETE from notes
WHERE id="${id}"
`);
});
};
const deleteAll = () => {
db.serialize(() => {
db.run("DROP TABLE IF EXISTS notes");
});
};
module.exports = {
getNotes,
getNoteById,
createNote,
updateNote,
deleteNote,
deleteAll,
};
Compared to the higher level abstraction of IOS’s CoreData framework, it was really nice just writing SQL queries.
Also as a general note, going back to untyped JavaScript was lovely.
I really enjoy TypeScript at work, and typed languages generally I think are a great way of cutting down on bugs, communicating design decisions with other developers, and generally making more robust, predictable software.
That said, for prototyping/individual projects where it is just me, I love the freedom that comes with raw untyped JavaScript.
Sure, I get runtime bugs, but I can fix them quickly.
Desktop Online/Offline
This was less smooth, and actually resulted in me starting to rethink my design of the online/offline stuff generally.
First up I needed the equivalent of IOS’s UserPreferences storage module, for storing any notes actions for later.
Because I’m back in my comfort zone with JavaScript/Node, I wrote my own way of storing JSON to a file locally in the place that Electron stores userData by default:
offline-updates.service.js
const { app } = require("electron");
const path = require("path");
const fs = require("fs");
const offlineUpdatesPath = path.join(
app.getPath("userData"),
"offline-updates.json"
);
const setUpdates = (updates) => {
console.log(`offline updates: ${updates}`);
fs.writeFileSync(offlineUpdatesPath, JSON.stringify(updates));
};
const getUpdates = () => {
if (fs.existsSync(offlineUpdatesPath)) {
return require(offlineUpdatesPath);
} else {
setUpdates([]);
return [];
}
};
const createNote = (note) => {
const updates = getUpdates();
updates.push(["createNote", note]);
setUpdates(updates);
};
const updateNote = (noteUpdate) => {
const updates = getUpdates();
updates.push(["updateNote", noteUpdate]);
setUpdates(updates);
};
const deleteNote = (noteId) => {
const updates = getUpdates();
updates.push(["deleteNote", noteId]);
setUpdates(updates);
};
const processOfflineUpdates = (socket) => {
getUpdates().forEach((update) => {
const action = update[0];
const payload = update[1];
console.log(
`processing offline update ${action}, with payload: ${payload}`
);
socket.emit(action, payload);
});
setUpdates([]);
};
module.exports = {
createNote,
updateNote,
deleteNote,
processOfflineUpdates,
};
So far so good.
Next step, how to figure out whether the user is online or not.
This is grosser 🙁
network-detector.service.js
const net = require("net");
let lastEmitted = false;
let _onChange;
const checkConnection = (onChange) => {
_onChange = onChange;
const connection = net.connect(
{
port: 80,
host: "google.com",
},
() => {
if (lastEmitted === false) {
lastEmitted = true;
_onChange(true);
}
}
);
connection.on("error", () => {
if (lastEmitted === true) {
lastEmitted = false;
_onChange(false);
}
});
};
const onNetworkChange = (onChange) => {
checkConnection(onChange);
setInterval(() => {
checkConnection(onChange, lastEmitted);
}, 5000);
};
module.exports = {
onNetworkChange,
};
Basically, every 5 seconds, try and fire a network event, if it works, you are online, otherwise you are not. In my notes service, I can subscribe to the events emitted from this service, and do the same if(online) style checks as in the IOS app.
The problem is the potential 5 second delay between being offline, and me knowing about it. This kind of breaks my solution as I could very easily try and send a bunch of stuff up to the server when I’m offline, and then just lose those actions completely.
At the end of last week, my thinking was that something like this might be the solution.
On the client side:
const updateNote = (prevNote, updatedNote) => {
let serverGotTheMessage = false;
const noteUpdate = {
id: updatedNote.id,
title: dmp.diff_main(prevNote.title, updatedNote.title),
body: dmp.diff_main(prevNote.body, updatedNote.body),
};
setTimeout(() => {
if (!serverGotTheMessage) {
noteStorage.updateNote(updatedNote);
offlineUpdates.updateNote(noteUpdate);
}
}, 1000);
socket.emit("updateNote", noteUpdate, () => {
serverGotTheMessage = true;
});
};
On the server side:
socket.on("updateNote", (payload, ack) => {
if (ack) {
ack();
}
debug(`updating ${userId} note ${payload.id}`);
updateNote(userId, payload, io);
});
I guess check back next week to see if that’s a good idea or not…
What next?
Figure out a more robust solution for offline/online updates.
As has been the case for the last 2 weeks, I really need to get the IOS app submitted to the App Store. Until I do that I can’t really move on from this project!
As part of that, I will need to set up separate production environments for my db, server and Auth0 stuff. Currently everything has been done on one environment.
If I have time, continue working on the Electron app, and figure out as soon as possible how best to distribute it/make installers etc. Focus on Mac OS for now.
The main priority is getting the IOS app done though. Wish me luck.