I misunderstood the model and wrote wrong information on the description.
There are some different models of USE in TensorFlow Hub.

To clarify differences, I checked the graph visualizations of USE models (universal-sentence-encoder-large/3 and universal-sentence-encoder/2) on TensorBoard.

https://tfhub.dev/google/universal-sentence-encoder-large/3

The model default output is <tf.Tensor 'module_apply_default/Encoder_en/hidden_layers/l2_normalize:0' shape=(2, 512) dtype=float32>.
It's understandable from the following graph:

https://tfhub.dev/google/universal-sentence-encoder/2

The model default output is <tf.Tensor 'module_apply_default/Encoder_en/hidden_layers/l2_normalize:0' shape=(?, 512) dtype=float32>.
The model is based on Deep Averaging Network, so the output of DNN is fed into hidden_layers:

Though I now almost understand the models, it's difficult to trace the detailed computation in the graphs (e.g., what is the concrete expression of kernel defined in tanh_layer_0 in universal-sentence-encoder-large/3?).

Can we access to TensorFlow codes that define the models in order to understand the models in detail?

The code that defines the models is not available. Only the modules in tensorflow hub.

"https://tfhub.dev/google/universal-sentence-encoder/2" first takes Σ_w Embed(w) / √sentence length and feds it into several DNN layers, the output of the DNN layer is used as the output embedding.

"https://tfhub.dev/google/universal-sentence-encoder-large/3" has a transformer encoder and it uses the average pooling over all token embeddings at the last transformer layer as the output embedding.