ReinforcementLearningBase.jl

ReinforcementLearningBase.jl (RLBase) provides some common constants, traits, abstractions and interfaces in developing reinforcement learning algorithms in Julia.

Basically, we defined the following two main concepts in reinforcement learning:

• AbstractPolicy
• AbstractEnv

Rewards of all players sum to a constant

No chance player in the environment. And the game is fully deterministic.

Usually used to describe extensive-form game. The environment contains a chance player and the corresponding probability is known. Therefore, prob(env, player=chance_player(env)) must be defined.

The action space of the environment may contains illegal actions

Total rewards of all players may be different in each step

Every player gets the same reward

The inner state of some players' observations may be different

All actions in the action space of the environment are legal

All players observe the same state

Environment contains chance player and the probability is unknown. Usually only a dummy action is allowed in this case.

Environment with the DynamicStyle of SEQUENTIAL must takes actions from different players one-by-one.

Environment with the DynamicStyle of SIMULTANEOUS must take in actions from some (or all) players at one time

We can get reward after each step

No chance player in the environment. And the game is stochastic. To help increase reproducibility, these environments should generally accept a AbstractRNG as a keyword argument. For some third-party environments, at least a seed is exposed in the constructor.

Only get reward at the end of environment

Rewards of all players sum to 0. A special case of [CONSTANT_SUM].

(env::AbstractEnv)(action, player=current_player(env))

Super type of all reinforcement learning environments.

TODO:

Describe how to model a reinforcement learning environment. TODO: need more investigation Ref: https://bair.berkeley.edu/blog/2019/12/12/mbpo/

• Analytic gradient computation
• Sampling-based planning
• Model-based data generation
• Value-equivalence prediction Model-based Reinforcement Learning: A Survey. Tutorial on Model-Based Methods in Reinforcement Learning

(π::AbstractPolicy)(env) -> action

Policy is the most basic concept in reinforcement learning. Unlike the definition in some other packages, here a policy is defined as a functional object which takes in an environment and returns an action.

Use it to represent the goal state

See the definition of information set

Use it to represent the internal state.

MultiAgent(n::Integer) -> MultiAgent{n}()

n must be ≥ 2.

Sometimes people from different field talk about the same thing with a different name. Here we set the Observation{Any}() as the default state style in this package.

See discussions here

A wrapper to treat each element as a sub-space which supports:

• Base.in
• Random.rand

In some cases, we may not be interested in the action/state space. One can return WorldSpace() to keep the interface consistent.

Make an independent copy of env,

Set the seed of internal rng

ActionStyle(env::AbstractEnv)

For environments of discrete actions, specify whether the current state of env contains a full action set or a minimal action set. By default the MINIMAL_ACTION_SET is returned.

ChanceStyle(env) = DETERMINISTIC

Specify which role the chance plays in the env. Possible returns are:

• STOCHASTIC. This is the default return.
• DETERMINISTIC
• EXPLICIT_STOCHASTIC
• SAMPLED_STOCHASTIC

Specify the defalt state style when calling state(env).

DynamicStyle(env::AbstractEnv) = SEQUENTIAL

Only valid in environments with a NumAgentStyle of MultiAgent. Determine whether the players can play simultaneously or not. Possible returns are:

• SEQUENTIAL. This is the default return.
• SIMULTANEOUS.

InformationStyle(env) = IMPERFECT_INFORMATION

Distinguish environments between PERFECT_INFORMATION and IMPERFECT_INFORMATION. IMPERFECT_INFORMATION is returned by default.

NumAgentStyle(env)

Number of agents involved in the env. Possible returns are:

• SINGLE_AGENT. This is the default return.
• [MultiAgent][@ref].

Specify whether we can get reward after each step or only at the end of an game. Possible values are STEP_REWARD (the default one) or TERMINAL_REWARD.

StateStyle(env::AbstractEnv)

Define the possible styles of state(env). Possible values are:

• Observation{T}. This is the default return.
• InternalState{T}
• Information{T}
• You can also define your customized state style when necessary.

Or a tuple contains several of the above ones.

This is useful for environments which provide more than one kind of state.

UtilityStyle(env::AbstractEnv)

Specify the utility style in multi-agent environments. Possible values are:

• GENERAL_SUM. The default return.
• ZERO_SUM
• CONSTANT_SUM
• IDENTICAL_UTILITY

action_space(env, player=current_player(env))

Get all available actions from environment. See also: legal_action_space

chance_player(env)

Only valid for environments with a chance player.

child(env::AbstractEnv, action)

Treat the env as a game tree. Create an independent child after applying action.

current_player(env)

Return the next player to take action. For Extensive Form Games, a chance player may be returned. (See also chance_player) For SIMULTANEOUS environments, a simultaneous player is always returned. (See also simultaneous_player).

is_terminated(env, player=current_player(env))

legal_action_space(env, player=current_player(env))

For environments of MINIMAL_ACTION_SET, the result is the same with action_space.

legal_action_space_mask(env, player=current_player(env)) -> AbstractArray{Bool}

Required for environments of FULL_ACTION_SET.

priority(π::AbstractPolicy, experience)

Usually used in offline policies.

Get the action distribution of chance player.

prob(π::AbstractPolicy, env, action)

Only valid for environments with discrete actions.

prob(π::AbstractPolicy, env) -> Distribution

Get the probability distribution of actions based on policy π given an env.

Reset the internal state of an environment

reward(env, player=current_player(env))

simultaneous_player(env)

Only valid for environments of SIMULTANEOUS style.

spectator_player(env)

Used in imperfect multi-agent environments.

state(env, style=[DefaultStateStyle(env)], player=[current_player(env)])

The state can be of any type. However, most neural network based algorithms assume an AbstractArray is returned. For environments with many different states provided (inner state, information state, etc), users need to provide style to declare which kind of state they want.

state_space(env, style=[DefaultStateStyle(env)], player=[current_player(env)])

Describe all possible states.

Call this function after writing your customized environment to make sure that all the necessary interfaces are implemented correctly and consistently.

update!(π::AbstractPolicy, experience)

Update the policy π with online/offline experience or parameters.

walk(f, env::AbstractEnv)

Call f with env and its descendants. Only use it with small games.