Search Results for « Delayed Feedback »

The multi-armed bandit (MAB) models have attracted significant research attention due to their applicability and effectiveness in various real-world scenarios such as resource allocation, online advertising, and dynamic pricing. As an important branch, the adversarial MAB problems with delayed feedback have been proposed and studied by many researchers recently where a conceptual adversary strategically selects the reward distributions associated with each arm to challenge the learning algorithm and the agent experiences a delay between taking an action and receiving the corresponding reward feedback. However, the existing models restrict the feedback to be generated from only one user, which makes models inapplicable to the prevailing scenarios of multiple users (e.g. ad recommendation for a group of users). In this paper, we consider that the delayed feedback results are from multiple users and are unrestricted on internal distribution. In contrast, the feedback delay is arbitrary and unknown to the player in advance. Also, for different users in a round, the delays in feedback have no assumption of latent correlation. Thus, we formulate an adversarial MAB problem with multi-user delayed feedback and design a modified EXP3 algorithm MUD-EXP3, which makes a decision at each round by considering the importance-weighted estimator of the received feedback from different users. On the premise of known terminal round index $T$, the number of users $M$, the number of arms $N$, and upper bound of delay $d_{max}$, we prove a regret of $\mathcal{O}(\sqrt{TM^2\ln{N}(N\mathrm{e}+4d_{max})})$. Furthermore, for the more common case of unknown $T$, an adaptive algorithm AMUD-EXP3 is proposed with a sublinear regret with respect to $T$. Finally, extensive experiments are conducted to indicate the correctness and effectiveness of our algorithms.

The influence of time delay in systems of two coupled excitable neurons is studied in the framework of the FitzHugh-Nagumo model. A time delay can occur in the coupling between neurons or in a self-feedback loop. The stochastic synchronization of instantaneously coupled neurons under the influence of white noise can be deliberately controlled by local time-delayed feedback. By appropriate choice of the delay time, synchronization can be either enhanced or suppressed. In delay-coupled neurons, antiphase oscillations can be induced for sufficiently large delay and coupling strength. The additional application of time-delayed self-feedback leads to complex scenarios of synchronized in-phase or antiphase oscillations, bursting patterns or amplitude death.

We analyze the stabilization of unstable steady states by delayed feedback control with a periodic time-varying delay in the regime of a high-frequency modulation of the delay. The average effect of the delayed feedback term in the control force is equivalent to a distributed delay in the interval of the modulation, and the obtained distribution depends on the type of the modulation. In our analysis we use a simple generic normal form of an unstable focus, and investigate the effects of phase-dependent coupling and the influence of the control loop latency on the controllability. In addition, we have explored the influence of the modulation of the delays in multiple delay feedback schemes consisting of two independent delay lines of Pyragas type. A main advantage of the variable delay is the considerably larger domain of stabilization in parameter space.

Time-delayed systems are found to display remarkable temporal patterns the dynamics of which split into regular and chaotic components repeating at the interval of a delay. This novel long-term behavior for delay dynamics results from strongly asymmetric nonlinear delayed feedback driving a highly damped harmonic oscillator dynamics. In the corresponding virtual space-time representation, the behavior is found to develop as a chimeralike state, a new paradigmatic object from the network theory characterized by the coexistence of synchronous and incoherent oscillations. Numerous virtual chimera states are obtained and analyzed, through experiment, theory, and simulations.

Dispatch in three-sided marketplaces provides a natural setting for reinforcement learning from world feedback: decisions are evaluated by delayed operational outcomes such as delivery speed, courier utilization, and merchant congestion. We present a deployed reinforcement learning system at DoorDash that adapts dispatch objective weights in a large-scale food-delivery marketplace using delayed signals. Rather than replacing the combinatorial assignment optimizer, a store-level policy learned from logged marketplace data selects a discrete multiplier that shifts the dispatch optimizer's tradeoff between delivery quality and batching efficiency. This interface enables offline policy learning under noisy, delayed, and coupled feedback while preserving production feasibility constraints and operational safeguards. We train a shared value function using centralized offline data and decentralized store-level execution, with Double Q-learning targets and a conservative regularizer to reduce out-of-distribution value overestimation. In a production switchback experiment, the offline-trained policy increases batching and reduces courier-side time costs without degrading customer-facing delivery quality. Results illustrate how world feedback from a live economic and logistics system can be used to safely adapt decision policies online.

Time-delayed feedback is exploited for controlling noise-induced motion in coherence resonance oscillators. Namely, under the proper choice of time delay, one can either increase or decrease the regularity of motion. It is shown that in an excitable system, delayed feedback can stabilize the frequency of oscillations against variation of noise strength. Also, for fixed noise intensity, the phenomenon of entrainment of the basic oscillation period by the delayed feedback occurs. This allows one to steer the time scales of noise-induced motion by changing the time delay.

We describe a flexible and modular delayed-feedback nonlinear oscillator that is capable of generating a wide range of dynamical behaviours, from periodic oscillations to high-dimensional chaos. The oscillator uses electrooptic modulation and fibre-optic transmission, with feedback and filtering implemented through real-time digital-signal processing. We consider two such oscillators that are coupled to one another, and we identify the conditions under which they will synchronize. By examining the rates of divergence or convergence between two coupled oscillators, we quantify the maximum Lyapunov exponents or transverse Lyapunov exponents of the system, and we present an experimental method to determine these rates that does not require a mathematical model of the system. Finally, we demonstrate a new adaptive control method that keeps two oscillators synchronized even when the coupling between them is changing unpredictably.

Online learning with delayed feedback has received increasing attention recently due to its several applications in distributed, web-based learning problems. In this paper we provide a systematic study of the topic, and analyze the effect of delay on the regret of online learning algorithms. Somewhat surprisingly, it turns out that delay increases the regret in a multiplicative way in adversarial problems, and in an additive way in stochastic problems. We give meta-algorithms that transform, in a black-box fashion, algorithms developed for the non-delayed case into ones that can handle the presence of delays in the feedback loop. Modifications of the well-known UCB algorithm are also developed for the bandit problem with delayed feedback, with the advantage over the meta-algorithms that they can be implemented with lower complexity.

Consider a scenario where a player chooses an action in each round $t$ out of $T$ rounds and observes the incurred cost after a delay of $d_{t}$ rounds. The cost functions and the delay sequence are chosen by an adversary. We show that in a non-cooperative game, the expected weighted ergodic distribution of play converges to the set of coarse correlated equilibria if players use algorithms that have "no weighted-regret" in the above scenario, even if they have linear regret due to too large delays. For a two-player zero-sum game, we show that no weighted-regret is sufficient for the weighted ergodic average of play to converge to the set of Nash equilibria. We prove that the FKM algorithm with $n$ dimensions achieves an expected regret of $O\left(nT^{\frac{3}{4}}+\sqrt{n}T^{\frac{1}{3}}D^{\frac{1}{3}}\right)$ and the EXP3 algorithm with $K$ arms achieves an expected regret of $O\left(\sqrt{\log K\left(KT+D\right)}\right)$ even when $D=\sum_{t=1}^{T}d_{t}$ and $T$ are unknown. These bounds use a novel doubling trick that, under mild assumptions, provably retains the regret bound for when $D$ and $T$ are known. Using these bounds, we show that FKM and EXP3 have no weighted-regret even for $d_{t}=O\left(t\log t\right)$. Therefore, algorithms with no weighted-regret can be used to approximate a CCE of a finite or convex unknown game that can only be simulated with bandit feedback, even if the simulation involves significant delays.

Online learning with delayed feedback has received increasing attention recently due to its several applications in distributed, web-based learning problems. In this paper we provide a systematic study of the topic, and analyze the effect of delay on the regret of online learning algorithms. Somewhat surprisingly, it turns out that delay increases the regret in a multiplicative way in adversarial problems, and in an additive way in stochastic problems. We give meta-algorithms that transform, in a black-box fashion, algorithms developed for the non-delayed case into ones that can handle the presence of delays in the feedback loop. Modifications of the well-known UCB algorithm are also developed for the bandit problem with delayed feedback, with the advantage over the meta-algorithms that they can be implemented with lower complexity.