Tim Dietrich

Custom Software Developer

Home Services Portfolio Blog About Contact Newsletter

EasySync: Beta 8 Now Available

I'm pleased to announce that EasySync "Beta 8" is now available for download. You can grab a copy from the EasySync Web site.

This is an important release for many reasons. One, there are a lot of significant changes in it, which we'll get to in just a moment. But perhaps most importantly, this is the last "beta" release of EasySync. The plan is to fix any bugs that are reported, and then in a week or so, officially release EasySync Version 1. So it was important that I get this release as close to perfect as possible.

Okay, so what's new? Well, a lot...

"Round-Tripping" Data During Syncs
In this version of EasySync, I made the difficult decision to "round trip" the data. This means that data that has been pushed up to the server during the "push" phase of the sync process is included in the payload that is received in the "pull" phase. I tried everything that I could think of to avoid having to do this, but it became clear that it was necessary. Here's why...

There are times when a user will want to sync from multiple devices via the same FileMaker account. In those cases, EasySync struggles to determine what to include in the pulled payload, because while it knows what data that user has pushed up to the server, it does not know what device it pushed it from. As a result, the user could end up not receiving data that they pushed from another device.

Also, if we don't "round trip" the data, there is the possibility that data will get out of sync, especially when conflicts arise during the push phase when the conflict resolution method being used is "server wins" or "last update wins." By "round tripping" the data, we get back the data as it truly is on the server, regardless of how any conflicts were resolved.

And so, "round tripping" the data became a necessity. I don't like it, but it seems like the best way to resolve these issues.

(One possible solution - and perhaps something you'll see in a future version of EasySync - is to add an additional EasySync field that will track the specific device that last updated a record.)

Support for Deletions
If you've been following along with EasySync's development, you know that I've struggled to come up with a good way to handle deleted records. The topic has come up in nearly every conversation that I've had with beta testers. I've said before that deletions make sync, which is already quite complicated, that much more complicated. And that's an understatement. Deletions make sync a nightmare. (Or at least they have the potential to.)

After spending a lot of time thinking about how to best handle deletions, and after running all kinds of crazy ideas past other developers, I think I've come up with a solution that works. Some of you will like it, and others won't. But here it is...

For databases that are being synced, you no longer let users delete records. Instead, you let them "flag" records for deletion. This behavior is implemented in both the mobile database and in the hosted database for users that connect directly to it. (You'll see a new "ES_Deleted" field that has been added to the syncing tables in the demo. That field is the "deleted flag" that I'm referring to.)

So a user that wants to delete a record "flags" the records. In the case of a mobile user, when they sync, that flag is sent to the server just like any other field.

On the mobile side, I've introduced a new phase to the sync process, and I'm calling it the "deletion" phase. It only runs if the "pull" phase is successful. During this new phase, EasySync looks for records that have been flagged for deletion and truly deletes them from the mobile device.

The nice thing about this approach is that other mobile users will also see the deletion flag come down when they sync, so they won't end up with deleted records on their devices. Additionally, it means that users have the opportunity to "undelete" records. A mobile user who has accidentally flagged a record for deletion, but hasn't synced yet, can simply "undelete" the record (clear the flag). In cases where records have been deleted from the mobile device, a developer or manager can log into the hosted database and "undelete" a record when necessary. The mobile users will see those undeleted records appear back on their devices after the next sync.

Note that, because we cannot do anything other than "SELECT" in ExecuteSQL calls, in order for EasySync to delete records from the mobile device, it needs to be able to find them first. To do that, it needs context - which means it also needs layouts. (In the example demo database, you'll see that I've added a few layouts in a new "EasySync Delete Support" layout folder. Those are needed to delete the survey document and hobby child records.) So if you are going to use EasySync's delete feature, you need to have layouts setup in your database that are based on your "ES_" table occurrences. I don't like having to add these additional layouts, but it is necessary. (Support for INSERT, UPDATE, and DELETE in ExecuteSQL can't come soon enough!)

So, is this a good way to handle deletions? I think so, but I also expect that some users won't like it because the "flagged for deletion" records will linger on the mobile devices until a sync occurs. On the hosted database, they will linger as well. You might resolve some of these issues using some ingenuity. For example, using permissions so that records that have the deleted flag set aren't visible. You could also setup a scheduled script on the server that runs every once in awhile and deletes flagged records that haven't been undeleted after a certain period of time. So you have some options.

Another point worth mentioning is that, if you don't like this approach, you can simply turn off the feature. There's a new setting in the mobile database that you can use to disable the "deletion phase" altogether.

In any case, EasySync now officially supports deletions.

Support for Repeating Fields
Yes, really. We have support for repeating fields. I've joked that repeating fields are to FileMaker what training wheels are to bikes. But apparently, people still use them. (Personally, I avoid them like the plague, but that's a conversation for another time.)

In any case, one of my unspoken goals with EasySync has been that "if you can get a value into a field, EasySync can sync it." So support for repeating fields has been added in Beta 8, and you'll see a new "Repeater" repeating field added to the Survey table that shows that repeating fields can be synced.

"Initial Pull" Example Script
The demo database also includes an example script that illustrates how a developer might choose to handle what I've been referring to as an "initial pull." This is the situation where a new user has never synced before, has been given a "starter database" with no data loaded into it, and is therefore very likely going to be pulling a much larger than usual payload when they perform their first sync.

The example script is named "Example - Initial Pull" and can be found in the "EasySync Implementation Examples" script folder. (Also be sure to see the "Prepare Payload for Client" script in the hosted database for some minor customizations that were made to support this example. )

In the example script, several payloads are pulled by calling "Pull Payload" and passing some additional info to the server. In this case, the additional info that is passed to the server is the range of survey takers (based on last names) that are to be included in the pulled payload. Note that you can certainly use other methods to pull records by ranges, with those ranges being based on your particular solution and the nature of the data being synced.

So, how well does this approach work? In a test with 5,000 records on the host, pulling them using the standard sync method took 11+ minutes. Using the method demonstrated in the "initial pull" script, that pull took 2 minutes and 30 seconds. That's a significant difference.

However, there is a point at which the "initial pull" simply doesn't make sense. For example, in a test with 500 records on the server, pulling them using the "initial pull" method took 20 seconds, while pulling them using the standard sync method took only 13 seconds. So if your initial pull payload isn't very large, you should probably stick with using the standard method.

"Partial Push" Example Script
The demo database now includes an example script that shows how a developer might choose to add support for pushing a subset of the local data (something that I have been referring to as a "partial push"). It is by no means perfect, but it works. You'll find the script, which is named "Example - Partial Push," in a new "EasySync Implementation Examples" script folder in the mobile database. The script is wired to a new "Sync This Survey" button, which is located in the header area of the "Survey Form - iPad" layout.

The script pushes a single survey, along with its related child "hobby" and "document" records, to the server without pushing the entire set of records that would otherwise be in the payload. It is ideal for cases where a user is in a hurry and needs to get one record (and its child records) posted to the server quickly, without needing to wait for an entire sync cycle to complete.

So, what are the problems with this approach?
1. After the partial push, the pushed records are still set in such a way that they will be pushed again during the next full sync.
2. This is essentially a "push" that happens without a corresponding "pull." As a result, it is possible that the pushed data won't be applied to the server (if there is a conflict, and depending on the conflict method being used), and the user won't know that because there was no "pull."
3. If the "push" is being done to process deleted records, then this partial push approach won't work as the user might expect it to. The pushed records be flagged for deletion on the server, but because the new "deletion" phase of the sync process has been bypassed, the records will remain on the local device (until the next full push).

Regardless of its shortcomings, this is one way to handle the partial pushes.

Performance Improvements When Processing Payloads
We're now searching for the closing "" tag starting at the end of the payload, which improves processing speed slightly. Thanks go out to Daniel Wood (Twitter: @FMweetbicks) for the suggestion.

Bug Fix: Params Passed for Ping, Push, and Pull
There was also one minor bug fix included in this release. The "Get ( SystemIPAddress )" parameter, which is included in the script parameter that gets passed to the server during the ping, push, and pull phases, is now treated as a comma-delimited list. This resolves an issue that arises when the mobile client is using a device that has multiple IP addresses assigned to it. Thanks to Paul Jansen of APJ Computing Solutions ( http://www.apjuk.com/ ) for reporting this one.

Okay, so there you have it. EasySync Beta 8. As always, thanks for your interest in EasySync, and please let me know what you think.