Overview

One of our testnet validators got app-hashed at height 5106385 that only included a contract execution for a wormhole transfer.

Background

When nodes crashed due to missing wasm files we saw the error below which is self-explanatory:

failed to execute message; message index: 0: Error calling the VM: Cache error: Error opening Wasm file for reading: No such file or directory (os error 2): instantiate wasm contract failed

This time we see a different error on this node by looking at the block results in the faulty block indicating absence of AVX-512 on the CPU:

*failed* to execute message; message index: 0: Error calling the VM: Error instantiating a Wasm module: Error instantiating module: CpuFeature(\\"EnumSet(AVX512DQ | AVX512VL | AVX512F)\\"): execute wasm contract failed

What we know

Contract compilation

• The contract was compiled with the cosmwasm-rust-optimizer as every other contract on testnet/mainnet based on this Dockerfile
• The CPU does not have AVX512 support as per this gist
• The contract was compiled on a GCP node with normal hardware specs

App-hashed node

• Running on AWS, t2.xlarge machine

• The CPU does not have AVX512 support as per this gist

• The OS is Ubuntu 22.04.01 LTS

• The node is not missing any wasm files

• The checksum from the wormhole contract in the wasm local state matches the checksum here

• The node is running the latest injectived binary as other consensus nodes that did not crash

  $ injectived version Version dev (bb597ee) Compiled at 20221116-1033 using Go go1.18.3 (amd64)