So I’ve been in talks with Fauna recently for…reasons…and one of the recommendations throughout our conversations was to explore the product and build something with it. Now I’ve worked with Fauna in the past but I admittedly built a wrapper around FQL so I didn’t have fully learn and understand yet another query language.
What I ended up doing is rebuilding some of the database bits to call Fauna instead of GuardianForge. So I wanted to dive into the calls that I replaced, what they look like for querying DynamoDB, and the rebuilt queries in FQL. The backend is built with Go, so all examples for this will be in Go.
Last thing I'll leave you with before diving in: I absolutely do not claim to be an expert in FQL yet, so I might have gotten some the explanations wrong. It's simply my understanding & observations by converting existing DynamoDB code into FQL.
Fetch The Latest 15 Builds
On the home page of GuardianForge is a list of the most recent builds for visitors to start exploring. To fetch this data in Dynamo, I have a Local Secondary Index created on the publishedOn field of the build record. This lets me perform range queries using the Unix time stamp representation of any date/time, and passing in a limit of 15 only grabs the most recent 15. This returns an array (slice) of Build Summaries, which contain all the meta information on a Build (the bulk of the build data is stored as a JSON in S3).
// dynamo.go
func FetchLatestBuilds() ([]dbModels.BuildSummary, error) {
// Here I'm setting up the initial connection to Dynamo so I can use
tableName := os.Getenv("TABLE_NAME")
context, err := db.MakeContext(nil, &tableName)
if err != nil {
return nil, errors.Wrap(err, "(FetchLatestBuilds) make context")
}
// Set a limit of 15 records to pull
var limit int64 = 15
// Scan from the bottom of the index (newest first)
scanIndexForward := false
// Build the query input parameters
params := &dynamodb.QueryInput{
KeyConditions: map[string]*dynamodb.Condition{
"entityType": {
ComparisonOperator: aws.String("EQ"),
AttributeValueList: []*dynamodb.AttributeValue{
{
S: aws.String("build"),
},
},
},
// publishedOn is a unix timestamp in the db, so passing in
// the current unix timestamp to filter out any builds
// that go public in the future
"publishedOn": {
ComparisonOperator: aws.String("LT"),
AttributeValueList: []*dynamodb.AttributeValue{
{
N: aws.String(fmt.Sprint(time.Now().Unix())),
},
},
},
},
// Here I'm specifying the index I want to scan, along with
// the other params declared above
IndexName: aws.String("idx_publishedOn"),
Limit: &limit,
TableName: context.TableName,
ScanIndexForward: &scanIndexForward,
}
// Make the DynamoDB Query API call
result, err := context.DynamoSvc.Query(params)
if err != nil {
return nil, err
}
summaries := []dbModels.BuildSummary{}
// Loop over the items and convert them from Dynamo objects to
// builds used through the system
for _, i := range result.Items {
b := dbModels.Build{}
err = dynamodbattribute.UnmarshalMap(i, &b)
if err != nil {
return nil, errors.Wrap(err, "(getLatestBuilds) Unmarshalling dynmamo results")
}
s, err := b.GetBuildSummary()
if err != nil {
return nil, errors.Wrap(err, "(getLatestBuilds) Get build summary")
}
summaries = append(summaries, *s)
}
return summaries, nil
}
Let’s take a look at the Fauna equivalent now. I created a “builds” collection in Fauna to store the same data as in Dynamo. I also needed to create an index in Fauna to pull this off but what was confusing is that it doesn’t seem that the web UI has feature parity with the CLI, so I couldn’t create an index sorted on a data field using the UI. Luckily the web portal also has an area where I can write queries directly, which was very helpful. The following script is what I used to create that index:
CreateIndex({
name: "builds_orderBy_publishedOn_desc2",
source: Collection("builds"),
values: [
{ field: ["data", "publishedOn"] },
{ field: ["ref"] }
]
})
Here is the article I used to help with this process: Sort with indexes
Once that was in place, I could perform the same query like so.
// FaunaProvider.go
// I needed to create a struct since all data in a Fauna collection
// is stored in the `data` field
type FaunaBuildRecord struct {
Build dbModels.Build `fauna:"data"`
}
// FaunaProvider is a type I created to hold the Fauna client
func (fp *FaunaProvider) FetchLatestBuilds() ([]dbModels.BuildSummary, error) {
// Every query to Fauna is wrapped in this Query call
result, err := fp.Client.Query(
// Take is used to pull only the first N items in an array
// Map is used to take a set of data and pass it
// into another function (Lambda in this case)
f.Take(15, f.Map(
// Paginate is used to paginate records (I'm not sure this
// needed to be honest)
f.Paginate(
// Match is used make a comparison. Our index has no terms defined so
// nothing needs to be matched
f.Match(
// Index specifies the index we created in the earlier step.
f.Index("builds_orderBy_publishedOn_desc"),
),
),
// Lambda lets you perform operations with the data within Fauna
// Arr is pulling data from the Map function and defining
// variables with the data that Fauna can work with
// Get & Var are used to pull the `ref` value of the item (similar to a
// primary key or ID in a database) and get the actual document
f.Lambda(f.Arr{"publishedOn", "ref"}, f.Get(f.Var("ref"))),
),
))
if err != nil {
return nil, errors.Wrap(err, "(FaunaProvider.FetchLatestBuilds) Execute query")
}
// Here Im parsing the data from the results into a struct I can work with.
var records []FaunaBuildRecord
err = result.At(f.ObjKey("data")).Get(&records)
if err != nil {
return nil, errors.Wrap(err, "(FaunaProvider.FetchLatestBuilds) result.Get")
}
// And here is where Im doing some final parsing on the data to make sure its good.
summaries := []dbModels.BuildSummary{}
for _, r := range records {
s, err := r.Build.GetBuildSummary()
if err != nil {
return nil, errors.Wrap(err, "(FaunaProvider.FetchLatestBuilds) Get build summary")
}
summaries = append(summaries, *s)
}
jbytes, err := json.Marshal(summaries)
if err != nil {
// Yes, I get creative with my errors in POC code 😅
return nil, errors.Wrap(err, "asdlfkjasldkjf")
}
str := string(jbytes)
log.Println(str)
return summaries, nil
}
Here is the part that queries Fauna without the comments, just to see what it looks like.
result, err := fp.Client.Query(
f.Take(15, f.Map(
f.Paginate(
f.Match(
f.Index("builds_orderBy_publishedOn_desc"),
),
),
f.Lambda(f.Arr{"publishedOn", "ref"}, f.Get(f.Var("ref"))),
),
))
Here are my final observations after the conversion:
- Although FQL is somewhat confusing when compared to working with Dynamo Query Params, the code itself is more concise when you remove all the comments I added (for better or worse).
- I needed to define the sort order when creating the index in FQL, whereas the index in Dynamo can be sorted when executing a query. It doesn't seem like Fauna charges for more indexes, so this isn't too big of a deal in my opinion.
- From my understanding, FQL executes all of the logic within the database engine, meaning you have more flexibility when querying the database, and queries can be optimized better to return only what is needed. This can be tricky to pull off in Dynamo as you generally have to fully understand your access patters to use Dynamo efficiently.
Saving a Build
This is used to accept build data via the API and save it to the database. Here is the code for Dynamo. This one is relatively straightforward. Dynamo by default upserts data, so if it finds a record in the database that matches your unique composite key (partition key & sort key combined), it updates the data instead of creating a new record.
// dynamo.go
func PutBuildToDynamo(awsSession *session.Session, buildRecord dbModels.Build) error {
svc := dynamodb.New(awsSession)
tableName := os.Getenv("TABLE_NAME")
buildRecord.EntityType = "build"
// Convert my Build struct into a Dynamo Item
item, err := dynamodbattribute.MarshalMap(buildRecord)
if err != nil {
return errors.Wrap(err, "(PutBuildToDynamo) Marshal map")
}
// Save the item
input := &dynamodb.PutItemInput{
Item: item,
TableName: aws.String(tableName),
}
_, err = svc.PutItem(input)
if err != nil {
return err
}
return nil
}
Fauna was a bit more complex to mimic this logic. I needed to create another index to be able to fetch a build by its ID first. This is mainly because I already have an ID attribute for builds and I’m not using the native Ref object that identifies a document in Fauna. Once I had the index, I could use a the If function with Exists to check if the Build exists in the collection first, then decide to either create it or update it.
You’ll also notice we’re using MatchTerm instead of Match. I’m not sure if its a Go thing since MatchTerm is used to compare against another value, but Match seems to be used elsewhere regardless if you need to compare with a value or not.
// FaunaProvider.go
func (fp *FaunaProvider) PutBuild(buildRecord dbModels.Build) error {
_, err := fp.Client.Query(
// If lets you execute conditionals within the db engine using an (condition, true, false) syntax
f.If(
// Check if the record with that Id exists
f.Exists(f.MatchTerm(f.Index("builds_byBuildId3"), buildRecord.Id)),
// If it does, run the Update function to update it
f.Update(
// We need to run a Select to get the Ref so we can update it
f.Select("ref", f.Get(f.MatchTerm(f.Index("builds_byBuildId3"), buildRecord.Id))),
// Replace the `data` field which holds all our data
f.Obj{
"data": buildRecord,
},
),
// If the record does not exist, Create it, passing in the Collection first, then Object second
f.Create(
f.Collection("builds"),
f.Obj{
"data": buildRecord,
},
),
),
)
if err != nil {
return errors.Wrap(err, "(FaunaProvider.PutBuild) Execute query")
}
return nil
}
Fetch Build By Id
Getting a single Build by ID is used all the time in GuardianForge. Here is the Dynamo code, relatively straightforward.
// dynamo.go
func FetchBuildById(buildId string) (*dbModels.Build, error) {
tableName := os.Getenv("TABLE_NAME")
context, err := db.MakeContext(nil, &tableName)
if err != nil {
return nil, errors.Wrap(err, "(FetchBuildsById) make context")
}
params := &dynamodb.GetItemInput{
TableName: context.TableName,
Key: map[string]*dynamodb.AttributeValue{
// entityType is my partition key, and its always `build` for a build
"entityType": {
S: aws.String("build"),
},
// entityId is the sort key, a simple GUID
"entityId": {
S: aws.String(buildId),
},
},
}
result, err := context.DynamoSvc.GetItem(params)
if err != nil {
return nil, errors.Wrap(err, "(FetchBuildsById) GetItem")
}
record := dbModels.Build{}
err = dynamodbattribute.UnmarshalMap(result.Item, &record)
if err != nil {
return nil, errors.Wrap(err, "(FetchBuildsById) Unmarshal Dynamo attribute")
}
return &record, nil
}
I needed to use an index again here because as I stated earlier, I’m using my own ID instead of a Fauna Ref. The key takeaway is the MatchTerm function only returned a Ref, and wrapping it in Get pulls the full document out of the collection.
// FaunaProvider.go
func (fp *FaunaProvider) FetchBuildById(buildId string) (*dbModels.Build, error) {
log.Println("(FaunaProvider.FetchBuildById) start")
result, err := fp.Client.Query(
// Get uses the Ref from MatchTerm to get the data in the record
f.Get(
f.MatchTerm(
f.Index("builds_byBuildId3"),
buildId,
),
),
)
if err != nil {
return nil, errors.Wrap(err, "(FaunaProvider.FetchBuildById) Execute query")
}
var record FaunaBuildRecord
err = result.Get(&record)
if err != nil {
return nil, errors.Wrap(err, "(FaunaProvider.FetchByBuildId) Get")
}
fmt.Println(record.Build)
return &record.Build, nil
}
Delete a Build
So far we’ve addressed all general database CRUD operations instead of delete, so just adding this for completeness. Again, its pretty straight forward for both Dynamo and Fauna.
Dynamo:
// dynamo.go
func DeleteBuildFromDynamo(awsSession *session.Session, buildId string) error {
svc := dynamodb.New(awsSession)
tableName := os.Getenv("TABLE_NAME")
input := &dynamodb.DeleteItemInput{
Key: map[string]*dynamodb.AttributeValue{
"entityType": {
S: aws.String("build"),
},
"entityId": {
S: aws.String(buildId),
},
},
TableName: aws.String(tableName),
}
_, err := svc.DeleteItem(input)
if err != nil {
return errors.Wrap(err, "(DeleteBuildFromDynamo) Delete Item")
}
return nil
}
Fauna:
// FaunaProvider.go
func (fp *FaunaProvider) DeleteBuild(buildId string) error {
log.Println("(FaunaProvider.DeleteBuild) start")
_, err := fp.Client.Query(
f.Delete(
f.Select("ref", f.Get(f.MatchTerm(f.Index("builds_byBuildId3"), buildId))),
),
)
if err != nil {
return errors.Wrap(err, "(FaunaProvider.DeleteBuildFromDynamo) Execute query")
}
return nil
}
Some Tips I’ve Gathered While Building for Fauna
Here are some things I’ve learned along the way:
- Queries in general return an array of
RefsandGetneeds to be used to get the actual data. - Using
MapandLambdacan help transform or filter data out BEFORE its returned to your code from Fauna’s engine. - Testing my queries in the Shell section of the web UI was incredibly helpful, and translated well into Go code.
My Final Thoughts
So when comparing the way to access these two database systems, its pretty clear that they both have very distinct approaches. Fauna wraps much of the query into a set of nested functions, whereas Dynamo leverages objects & parameters passed into a small set of core functions.
I do like that Fauna seems to be more flexible than Dynamo while providing similar benefits, but you cant deny that the sheer scale of Dynamo and backing of Amazon makes it hard to beat from a support perspective. I don’t claim to be an expert in either, but I definitely will continue to use both going forward depending in the needs of the projects I'm building.