1. Q&A with Rich#
Is there any guidance for tile coding design? Does the more tilings achieve better decision performance?
Rich: It is about features selection, you need spend time to do this, the more important feature is, the more efficient algorithm achieves.
Why the evolution algorithm didn’t work well for tile coding design?
Rich: Evolution algorithm always died with nothing left, It is nothing to do with intelligence.
2. Tile Coding Implementation#
今天实现了一下Tile Coding，然后用Q_learning以及Sarsa($\lambda$)在Acrobot任务上进行了测试。这里介绍一下基于均匀网格的tile coding:
创建一组多组均匀分布网格来平滑地编码一个连续空间
def create_tiling_grid(low, high, bins=(10, 10), offsets=(0.0, 0.0)): """Define a uniformly-spaced grid that can be used for tile-coding a space. Parameters ---------- low : array_like Lower bounds for each dimension of the continuous space. high : array_like Upper bounds for each dimension of the continuous space. bins : tuple Number of bins or tiles along each corresponding dimension. For example, bins=(10, 10, ..., 10) means each state dimension is split into 10 intervals. offsets : tuple Per-dimension shift applied to the split points. Different offsets create overlapping tilings, which makes tile coding smoother than one fixed discretization grid. Returns ------- grid : list of array_like A list of arrays containing split points for each dimension. """ low = np.array(low) high = np.array(high) step_sizes = (high - low) / bins return [ np.arange(1, bins[dim]) * step_sizes[dim] + low[dim] + offsets[dim] for dim in range(len(low)) ] def create_tilings(low, high, tiling_specs): """Define multiple tilings using the provided specifications. Parameters ---------- low : array_like Lower bounds for each dimension of the continuous space. high : array_like Upper bounds for each dimension of the continuous space. tiling_specs : list of tuples A sequence of (bins, offsets) pairs. Each pair defines one tiling. More tilings create more active features per state-action pair and usually smoother value estimates, but each update becomes more costly. Returns ------- tilings : list A list of tilings (grids), each produced by create_tiling_grid(). """ return [ create_tiling_grid(low, high, bins, offsets) for bins, offsets in tiling_specs ]获取采样点的网格编码
def discretize(sample, grid): """Discretize a sample as per given grid. Parameters ---------- sample : array_like A single sample from the original continuous space. grid : list of array_like A list of arrays containing split points for each dimension. The returned integer for each dimension is the bin index into this grid. Returns ------- discretized_sample : tuple[int, ...] One integer bin index for each state dimension. """ return tuple( int(np.searchsorted(grid[dim], sample[dim], side="right")) for dim in range(len(sample)) ) def tile_encode(sample, tilings, flatten=False): """Encode a continuous state with all tilings. Parameters ---------- sample : array_like A single continuous state. tilings : list A list of tiling grids, each produced by create_tiling_grid(). flatten : bool If true, concatenate the encoded tile coordinates. The main algorithm keeps this false because it needs one coordinate tuple per tiling. Returns ------- encoded_sample : list[tuple[int, ...]] or np.ndarray Tile coordinates for every tiling. """ encoded = [discretize(sample, tiling) for tiling in tilings] if flatten: encoded = np.concatenate(encoded) return encoded