Many studies on aspect-based sentiment analysis (ABSA) aim to directly predict aspects and polarities at sentence level. However, it is not rare that a long sentence expresses multiple aspects. In this paper, we propose to study ABSA at EDU-level. Elementary discourse unit (EDU) in rhetorical structure theory is an atomic semantic unit, similar to a clause in a sentence. Through manual annotation of 8,823 EDUs, obtained from the SemEval-2014 Task 4 Restaurant Review dataset, we show that more than 97% of EDUs express at most one aspect. Based on this observation, we propose an EDU-level Capsule network for ABSA. EDU-Capsule learns EDU representations within its sentential context for aspect detection and sentiment prediction. EDU-Capsule outperforms strong baselines in our experiments on two benchmark datasets. Both the EDU-level annotations and EDU-Capsule source code are released to support further studies in this area.