Avoiding StackOverflowError at deserialization – Java I/O: Context-Specific Deserialization Filters

132. Avoiding StackOverflowError at deserialization

Let’s consider the following snippet of code:

// ‘mapOfSets’ is the object to serialize/deserialize
HashMap<Set, Integer> mapOfSets = new HashMap<>();
Set<Set> set = new HashSet<>();
mapOfSets.put(set, 1);
set.add(set);

And, we plan to serialize the mapOfSets object as follows (I assume that Converters.objectToBytes() is well-known from the previous problems):

byte[] mapSer = Converters.objectToBytes(mapOfSets);

Everything works just fine until we try to deserialize mapSer. At that moment, instead of a valid object, we will get a StackOverflowError as follows:

Exception in thread “main” java.lang.StackOverflowError
  at java.base/java.util.HashMap$KeyIterator
     .<init>(HashMap.java:1626)
  at java.base/java.util.HashMap$KeySet
     .iterator(HashMap.java:991)
  at java.base/java.util.HashSet
     .iterator(HashSet.java:182)
  at java.base/java.util.AbstractSet
     .hashCode(AbstractSet.java:120)
  at java.base/java.util.AbstractSet
     .hashCode(AbstractSet.java:124)
  …

The deserialization process got stuck in the hashCode() method of Set. The solution is to create a filter that will reject deserialization if the object has a graph depth bigger than 2. This can be a pattern-based filter as follows:

ObjectInputFilter filter = ObjectInputFilter.Config
  .createFilter(“maxdepth=2;java.base/*;!*”);

Next, call the deserialization process with this filter:

HashMap mapDeser = (HashMap) Converters
  .bytesToObject(mapSer, filter);      

I assume that Converters.bytesToObject() is well-known from the previous problems. This time, instead of getting StackOverflowError, the deserialization is rejected by the filter.

133. Avoiding DoS attacks at deserialization

Denial-of-service (DoS) attacks are typically malicious actions meant to trigger in a short period of time a lot of requests to a server, application, and so on. Generally speaking, a DoS attack is any kind of action that intentionally/accidentally overwhelms a process and forced it to slow down or even to crush. Let’s see a snippet of code that is a good candidate for representing a DoS attack at the deserialization phase:

ArrayList<Object> startList = new ArrayList<>();
List<Object> list1 = startList;
List<Object> list2 = new ArrayList<>();
for (int i = 0; i < 101; i++) {
  List<Object> sublist1 = new ArrayList<>();
  List<Object> sublist2 = new ArrayList<>();
          
  sublist1.add(“value: ” + i);
  list1.add(sublist1);
  list1.add(sublist2);
  list2.add(sublist1);
  list2.add(sublist2);
  list1 = sublist1;
  list2 = sublist2;
}

And, we plan to serialize the startList object as follows (I assume that Converters.objectToBytes() is well-known from the previous problems):

byte[] startListSer = Converters.objectToBytes(startList); 

Everything works just fine until we try to deserialize startListSer. At that moment, instead of a valid object, we will get … nothing! Actually, the application starts normally, but it just hanging there in the deserialization phase. The system slows down and after a while, it will eventually crush.The object graph is too deep to be deserialized and that leads to a behavior similar to a DoS attack. The solution is to create a filter that will reject deserialization if the object has a graph depth bigger than a safe value. This can be a pattern-based filter as follows:

ObjectInputFilter filter = ObjectInputFilter.Config
  .createFilter(“maxdepth=10;java.base/*;!*”);

Next, call the deserialization process with this filter:

ArrayList startListDeser = (ArrayList)
  Converters.bytesToObject(startListSer, filter);              

I assume that Converters.bytesToObject() is well-known from the previous problems. This time, the deserialization is rejected by the filter and the DoS attack is prevented.