Unmarshal remaining JSON after performing custom unmarshalling

Issue

I have a JSON object That contains an implementation of an interface within it. I’m attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.

I’ve managed to get it to implement the interface with a custom JSON unmarshal function however I’m struggling to piece together how to then marshal the rest of the fields

I’ve created an example in the Go playground

https://play.golang.org/p/ztF7H7etdjM

My JSON being passed into my application is

{

   "address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
   "nonce":13,
   "network_id":"qadre.demo.balance",
   "challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
   "type":"verifying_key",
   "verifying_key":{
      "verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
      "fqdn":"huski.service.key"
   },
   "signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
   "fqdn":"huski.service.SingleKeyProof"
}

I’ve attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes

The best solution I’ve come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though

var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
    return err
}

ad, ok := m["address"]
if ok {
    s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
    s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
    s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
    s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
    s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
    s.Signature = []byte(sig.(string))
}

Solution

Looking at what you’ve done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:

type WrappedSingleKey struct {
    FQDN         string          `json:"fqdn"`
    Address      string          `json:"address"`
    Nonce        int64           `json:"nonce"`
    Challenge    []byte          `json:"challenge"`
    NetworkID    string          `json:"network_id"`
    Type         string          `json:"type"`
    VerifyingKey json.RawMessage `json:"verifying_key"`
    OtherKey     json.RawMessage `json:"other_key"`
    Signature    []byte          `json:"signature"`
}

type SingleKey struct {
    FQDN         string     `json:"fqdn"`
    Address      string     `json:"address"`
    Nonce        int64      `json:"nonce"`
    Challenge    []byte     `json:"challenge"`
    NetworkID    string     `json:"network_id"`
    Type         string     `json:"type"`
    VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
    OtherType    *OtherKey  `json:"other_key,omitempty"`
    Signature    []byte     `json:"signature"`
}

So I’ve changed the type of your VerifyingKey field to a json.RawMessage. That’s basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.

In the unwrapped type, I’ve changed VerifyingKey to a pointer and added the omitempty bit to the tag. That’s just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:

func (s *SingleKey) UnmarshalJSON(data []byte) error {
    w := WrappedSingleKey{} // create wrapped instance
    if err := json.Unmarshal(data, &w); err != nil {
        return err
    }
    switch w.Type {
    case "verifying_key":
       var pk PublicKey
       if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
           return err
       }
       s.VerifyingKey = &pk // assign
    case "other_key":
        var ok OtherKey
        if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
            return err
        }
        s.OtherKey = &ok
    }
    // copy over the fields that didn't require anything special
    s.FQDN = w.FQDN
    s.Address = w.Address
}

This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It’s something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.

You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields… It’s up to you, really.

Note

I didn’t test this code, just wrote it as I went along. It’s something I’ve used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)

Answered By – Elias Van Ootegem

Answer Checked By – Katrina (GoLangFix Volunteer)

Leave a Reply

Your email address will not be published.